2024-03-16T11:56:52.737 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T11:56:52.737 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} --name pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434 --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/orig.config.yaml 2024-03-16T11:56:52.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T11:56:52.889 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} email: pdonnell@redhat.com first_in_suite: false job_id: '7602434' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwWd8rBTHCIAQBEQi5HXxWn8S2bWvP1M405ygUBRyMEBjX1B095iiaWEDws2QLjTsUg2K4LqmgJmcr/5vhlN7U= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEPN8BI7vA1jD+GZIPR75ZCizztI2zUCeQIKFnkor63kh2dMpO8kJc4da7TqQdaovLvcPKIWXe7OOdRFsKypyE= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYtuJvBvi2iHHwcVzQkrsBbGU4azcveJmHAbJYTZku90xWNJ0CCR9f4622ZojTPmiAex4NUwRK3sUfhxq18BFg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_13:00:41 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T11:56:52.889 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa; will attempt to use it 2024-03-16T11:56:52.890 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks 2024-03-16T11:56:52.890 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T11:56:52.892 INFO:teuthology.task.internal:Checking packages... 2024-03-16T11:56:52.915 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-16T11:56:52.915 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T11:56:52.915 INFO:teuthology.packaging:ref: None 2024-03-16T11:56:52.915 INFO:teuthology.packaging:tag: None 2024-03-16T11:56:52.915 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-16T11:56:52.915 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:56:52.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-16T11:56:53.074 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-16T11:56:53.075 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T11:56:53.082 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T11:56:53.082 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T11:56:53.090 INFO:teuthology.task.internal:Saving configuration 2024-03-16T11:56:53.110 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T11:56:53.115 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T11:56:53.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.815087', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwWd8rBTHCIAQBEQi5HXxWn8S2bWvP1M405ygUBRyMEBjX1B095iiaWEDws2QLjTsUg2K4LqmgJmcr/5vhlN7U='} 2024-03-16T11:56:53.158 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.817083', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEPN8BI7vA1jD+GZIPR75ZCizztI2zUCeQIKFnkor63kh2dMpO8kJc4da7TqQdaovLvcPKIWXe7OOdRFsKypyE='} 2024-03-16T11:56:53.178 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.818824', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYtuJvBvi2iHHwcVzQkrsBbGU4azcveJmHAbJYTZku90xWNJ0CCR9f4622ZojTPmiAex4NUwRK3sUfhxq18BFg='} 2024-03-16T11:56:53.178 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T11:56:53.184 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T11:56:53.184 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T11:56:53.184 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T11:56:53.184 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T11:56:53.286 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa9b457a160>, signals=[15]) 2024-03-16T11:56:53.286 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T11:56:53.293 INFO:teuthology.task.internal:Opening connections... 2024-03-16T11:56:53.293 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T11:56:53.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:56:53.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-16T11:56:53.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:56:53.472 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-16T11:56:53.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:56:53.563 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T11:56:53.570 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-16T11:56:53.588 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-16T11:56:53.589 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:56:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:56:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:56:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:56:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:56:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:56:53.647 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-16T11:56:53.703 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-16T11:56:53.721 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-16T11:56:53.722 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-16T11:56:53.776 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-16T11:56:53.776 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-16T11:56:53.776 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-16T11:56:53.776 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:56:53.776 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:56:53.777 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:56:53.777 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-16T11:56:53.806 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-16T11:56:53.824 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-16T11:56:53.824 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:56:53.882 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:56:53.883 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:56:53.883 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-16T11:56:53.915 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T11:56:53.925 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T11:56:53.932 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T11:56:53.932 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:56:53.934 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:56:53.936 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:56:53.952 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T11:56:53.958 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T11:56:53.959 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:56:53.990 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:56:53.993 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:56:54.089 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T11:56:54.168 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T11:56:54.183 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-16T11:56:54.183 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-16T11:56:54.183 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-16T11:56:54.184 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T11:56:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:56:54.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-16T11:56:54.519 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-16T11:56:54.519 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-16T11:56:54.519 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-16T11:56:54.537 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-16T11:56:54.537 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-16T11:56:54.537 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-16T11:56:54.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum upgrade -y linux-firmware 2024-03-16T11:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-16T11:56:58.376 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-16T11:57:00.633 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-03-16T11:57:02.318 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-16T11:57:07.327 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-16T11:57:08.778 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 15 kB 00:00 2024-03-16T11:57:09.771 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-16T11:57:14.721 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-16T11:57:17.040 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-16T11:57:17.055 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 1 Package 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 375 M 2024-03-16T11:57:17.056 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-16T11:57:23.880 INFO:teuthology.orchestra.run.smithi098.stdout:linux-firmware-20240219-143.el9.noarch.rpm 55 MB/s | 375 MB 00:06 2024-03-16T11:57:23.916 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:57:23.916 INFO:teuthology.orchestra.run.smithi098.stdout:Total 55 MB/s | 375 MB 00:06 2024-03-16T11:57:23.916 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-16T11:57:23.960 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-16T11:57:23.960 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-16T11:57:26.417 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-16T11:57:26.418 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-16T11:57:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-16T11:57:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-16T11:57:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-16T11:57:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-16T11:57:34.743 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-16T11:57:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-16T11:57:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:57:35.127 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-16T11:57:35.127 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-16T11:57:35.128 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:57:35.128 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-16T11:57:35.417 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-16T11:57:35.417 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-16T11:57:35.571 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-16T11:57:35.571 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-16T11:57:35.574 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi098.front.sepia.ceph.com to complete... 2024-03-16T11:57:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16 11:57:47 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-16T11:57:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:FINISHED --2024-03-16 11:57:47-- 2024-03-16T11:57:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:Total wall clock time: 12s 2024-03-16T11:57:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:Downloaded: 1 files, 321M in 11s (28.5 MB/s) 2024-03-16T11:57:47.195 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-16T11:57:47.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-16T11:57:49.125 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying... ######################################## 2024-03-16T11:57:49.184 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing... ######################################## 2024-03-16T11:57:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:Updating / installing... 2024-03-16T11:57:56.775 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-16T11:58:13.535 INFO:teuthology.orchestra.run.smithi098.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T11:58:36.010 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-16T11:58:36.010 DEBUG:teuthology.orchestra.run.smithi098:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-16T11:58:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-16T11:58:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-16T11:58:36.081 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-16T11:58:36.082 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-16T11:58:36.083 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-16T11:58:36.084 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-16T11:58:36.085 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-16T11:58:36.086 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-16T11:58:36.087 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-16T11:58:36.088 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-16T11:58:36.089 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-16T11:58:36.090 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-16T11:58:36.091 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-16T11:58:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-16T11:58:36.093 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-16T11:58:36.094 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-16T11:58:36.095 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-16T11:58:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-16T11:58:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-16T11:58:36.098 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-16T11:58:36.099 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-16T11:58:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-16T11:58:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-16T11:58:36.102 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-16T11:58:36.103 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-16T11:58:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-16T11:58:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-16T11:58:36.106 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-16T11:58:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-16T11:58:36.108 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-16T11:58:36.109 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-16T11:58:36.110 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-16T11:58:36.111 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-16T11:58:36.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-16T11:58:36.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-16T11:58:36.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-16T11:58:36.113 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-16T11:58:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-16T11:58:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-16T11:58:36.116 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-16T11:58:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-16T11:58:36.118 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-16T11:58:36.119 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-16T11:58:36.120 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-16T11:58:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-16T11:58:36.122 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-16T11:58:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-16T11:58:36.124 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-16T11:58:36.125 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-16T11:58:36.126 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-16T11:58:36.127 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-16T11:58:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-16T11:58:36.129 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-16T11:58:36.130 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-16T11:58:36.131 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-16T11:58:36.132 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-16T11:58:36.133 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-16T11:58:36.134 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-16T11:58:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-16T11:58:36.136 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-16T11:58:36.137 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-16T11:58:36.138 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-16T11:58:36.139 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-16T11:58:36.140 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-16T11:58:36.141 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-16T11:58:36.142 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-16T11:58:36.143 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-16T11:58:36.144 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-16T11:58:36.145 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-16T11:58:36.146 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-16T11:58:36.147 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-16T11:58:36.148 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-16T11:58:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-16T11:58:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-16T11:58:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-16T11:58:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-16T11:58:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-16T11:58:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-16T11:58:36.155 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-16T11:58:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-16T11:58:36.157 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-16T11:58:36.158 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-16T11:58:36.159 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-16T11:58:36.160 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-16T11:58:36.161 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-16T11:58:36.162 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-16T11:58:36.163 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-16T11:58:36.164 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-16T11:58:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-16T11:58:36.166 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-16T11:58:36.167 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-16T11:58:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-16T11:58:36.169 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-16T11:58:36.170 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-16T11:58:36.171 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-16T11:58:36.172 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-16T11:58:36.173 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-16T11:58:36.174 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-16T11:58:36.175 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-16T11:58:36.176 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-16T11:58:36.177 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-16T11:58:36.178 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-16T11:58:36.179 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-16T11:58:36.180 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-16T11:58:36.181 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-16T11:58:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-16T11:58:36.183 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-16T11:58:36.184 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-16T11:58:36.185 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-16T11:58:36.186 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-16T11:58:36.187 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-16T11:58:36.188 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-16T11:58:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-16T11:58:36.190 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-16T11:58:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-16T11:58:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-16T11:58:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-16T11:58:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-16T11:58:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-16T11:58:36.196 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-16T11:58:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-16T11:58:36.198 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-16T11:58:36.199 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-16T11:58:36.200 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-16T11:58:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-16T11:58:36.202 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-16T11:58:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-16T11:58:36.204 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-16T11:58:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-16T11:58:36.206 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-16T11:58:36.207 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-16T11:58:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-16T11:58:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-16T11:58:36.210 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-16T11:58:36.211 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-16T11:58:36.212 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-16T11:58:36.213 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-16T11:58:36.214 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-16T11:58:36.215 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-16T11:58:36.216 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-16T11:58:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-16T11:58:36.218 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-16T11:58:36.219 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-16T11:58:36.220 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-16T11:58:36.221 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-16T11:58:36.222 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-16T11:58:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-16T11:58:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-16T11:58:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-16T11:58:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-16T11:58:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-16T11:58:36.228 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-16T11:58:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-16T11:58:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-16T11:58:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-16T11:58:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-16T11:58:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-16T11:58:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-16T11:58:36.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-16T11:58:36.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-16T11:58:36.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-16T11:58:36.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-16T11:58:36.231 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-16T11:58:36.232 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-16T11:58:36.233 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-16T11:58:36.234 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-16T11:58:36.235 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-16T11:58:36.236 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-16T11:58:36.237 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-16T11:58:36.238 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-16T11:58:36.239 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-16T11:58:36.240 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-16T11:58:36.241 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-16T11:58:36.242 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-16T11:58:36.243 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-16T11:58:36.244 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-16T11:58:36.245 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-16T11:58:36.246 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-16T11:58:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-16T11:58:36.248 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-16T11:58:36.249 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-16T11:58:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-16T11:58:36.251 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-16T11:58:36.252 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-16T11:58:36.253 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-16T11:58:36.257 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:36.258 DEBUG:teuthology.orchestra.run.smithi098:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:postinstall scriptlet (using /bin/sh): 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:for file in vmlinuz System.map config; do 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: fi 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:done 2024-03-16T11:58:36.281 INFO:teuthology.orchestra.run.smithi098.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:36.282 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2024-03-16T11:58:36.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-16T11:58:36.367 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T11:58:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T11:58:36.369 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T11:58:36.370 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T11:58:36.371 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:36.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T11:58:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-16T11:58:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T11:58:38.420 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-16T11:58:38.423 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-16T11:58:38.438 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.CKahTIAZzN 2024-03-16T11:58:38.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CKahTIAZzN 2024-03-16T11:58:38.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.CKahTIAZzN 2024-03-16T11:58:38.647 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.CKahTIAZzN is 6KB 2024-03-16T11:58:38.695 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.CKahTIAZzN 2024-03-16T11:58:38.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T11:58:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T11:58:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T11:58:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T11:58:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-16T11:58:38.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-16T11:58:38.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-16T11:59:08.946 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-16T11:59:08.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T11:59:08.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:59:27.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T11:59:36.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T11:59:36.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:59:39.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T11:59:51.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T11:59:51.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T12:00:09.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-16T12:00:24.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T12:00:24.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T12:00:28.321 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-16T12:00:28.984 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T12:00:28.984 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-16T12:00:28.985 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-16T12:00:29.022 INFO:teuthology.orchestra.run.smithi098.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-16T12:00:29.023 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-16T12:00:29.023 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T12:00:29.023 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T12:00:30.024 DEBUG:teuthology.parallel:result is None 2024-03-16T12:00:30.024 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T12:00:30.032 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T12:00:30.032 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T12:00:30.040 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T12:00:30.042 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T12:00:30.083 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T12:00:30.089 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T12:00:30.129 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T12:00:30.129 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T12:00:30.134 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T12:00:30.136 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T12:00:30.247 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T12:00:30.254 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T12:00:30.254 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T12:00:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:00:30.271 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T12:00:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:00:30.287 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T12:00:30.292 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T12:00:30.313 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T12:00:30.367 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.375 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.391 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.399 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.424 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.505 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T12:00:30.509 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T12:00:30.516 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T12:00:30.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T12:00:30.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T12:00:30.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T12:00:30.614 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T12:00:30.623 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T12:00:30.624 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T12:00:30.628 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T12:00:30.630 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T12:00:30.668 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:30.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:30.864 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:30.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:31.008 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:00:31.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T12:00:31.105 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:31.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:31.206 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:31.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:31.333 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:00:31.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T12:00:31.402 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:31.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T12:00:31.779 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:31.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T12:00:31.868 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:00:31.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T12:00:31.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-16T12:00:31.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-16T12:00:31.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-16T12:00:31.988 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:00:32.017 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:00:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:00:32.329 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T12:00:32.337 INFO:teuthology.task.internal:Starting timer... 2024-03-16T12:00:32.337 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T12:00:32.378 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T12:00:32.402 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-16T12:00:32.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-16T12:00:32.511 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-16T12:00:32.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-16T12:00:32.570 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-16T12:00:32.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-16T12:00:32.620 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-16T12:00:32.622 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T12:00:32.622 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-16T12:00:32.705 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-16T12:00:32.705 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-16T12:00:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-16T12:00:32.743 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-16T12:00:32.769 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-16T12:00:32.769 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-16T12:00:32.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:00:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:00:32.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:00:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:00:32.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:00:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:00:32.903 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T12:00:32.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-16T12:00:32.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-16T12:00:33.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-16T12:00:33.063 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T12:00:33.073 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-16T12:00:33.095 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-16T12:00:33.277 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T12:00:33.360 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T12:00:33.362 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi153.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T12:06:22.737 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-16T12:06:22.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T12:06:22.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T12:06:22.884 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-16T12:06:23.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-16T12:06:23.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T12:06:23.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T12:06:23.120 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-16T12:06:23.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-16T12:06:23.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-16T12:06:23.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T12:06:23.278 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-16T12:06:23.351 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-16T12:06:23.351 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T12:06:23.364 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T12:06:23.364 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T12:06:23.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:06:23.371 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T12:06:23.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:06:23.373 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T12:06:23.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:06:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T12:06:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T12:06:23.425 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T12:06:23.443 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T12:06:23.449 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-16T12:06:23.463 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-16T12:06:23.464 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T12:06:23.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-16T12:06:23.479 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T12:06:23.487 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-16T12:06:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T12:06:23.504 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T12:06:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T12:06:23.520 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T12:06:23.539 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:06:23.542 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:06:23.543 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-16T12:06:23.543 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.543 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.543 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.543 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.564 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.567 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-16T12:06:23.648 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-16T12:06:23.701 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T12:06:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T12:06:23.888 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.896 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:06:23.898 INFO:teuthology.run_tasks:Running task install... 2024-03-16T12:06:23.906 DEBUG:teuthology.task.install:project ceph 2024-03-16T12:06:23.907 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-16T12:06:23.907 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-16T12:06:23.907 INFO:teuthology.task.install:Using flavor: default 2024-03-16T12:06:23.912 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T12:06:23.913 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T12:06:23.913 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T12:06:23.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:06:23.916 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T12:06:23.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:06:23.917 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T12:06:23.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:06:24.065 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T12:06:24.065 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T12:06:24.071 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T12:06:24.071 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T12:06:24.074 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T12:06:24.074 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T12:06:24.176 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T12:06:24.177 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:06:24.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T12:06:24.178 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T12:06:24.179 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:06:24.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T12:06:24.208 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T12:06:24.208 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T12:06:24.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T12:06:24.278 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T12:06:24.278 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T12:06:24.301 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T12:06:24.354 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:06:24.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T12:06:24.384 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T12:06:24.384 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T12:06:24.395 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T12:06:24.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T12:06:24.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T12:06:24.463 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-16T12:06:24.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-16T12:06:24.535 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T12:06:24.578 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T12:06:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-16T12:06:24.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-16T12:06:24.663 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-16T12:06:24.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-16T12:06:24.700 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-16T12:06:24.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T12:06:24.840 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-16T12:06:24.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T12:06:25.011 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-16T12:06:25.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T12:06:25.448 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 193 kB/s | 83 kB 00:00 2024-03-16T12:06:25.479 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 257 kB/s | 83 kB 00:00 2024-03-16T12:06:25.898 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-16T12:06:25.946 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-16T12:06:26.118 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 147 kB/s | 83 kB 00:00 2024-03-16T12:06:26.289 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-16T12:06:26.307 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-16T12:06:26.793 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-16T12:06:26.824 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-16T12:06:26.909 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-16T12:06:27.369 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-16T12:06:28.022 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-16T12:06:28.266 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-16T12:06:28.543 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-16T12:06:29.582 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-03-16T12:06:29.673 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-16T12:06:31.053 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.0 MB 00:01 2024-03-16T12:06:31.408 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-16T12:06:35.626 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-16T12:06:36.012 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-16T12:06:37.126 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-03-16T12:06:38.450 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 858 kB/s | 8.0 MB 00:09 2024-03-16T12:06:40.518 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-16T12:06:41.013 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.1 MB 00:01 2024-03-16T12:06:42.980 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-03-16T12:06:44.587 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-16T12:06:44.993 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:00 2024-03-16T12:06:46.586 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-16T12:06:47.529 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-16T12:06:50.382 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-16T12:06:52.537 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-16T12:06:53.118 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:06:53.119 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:06:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T12:06:53.142 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T12:06:53.143 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T12:06:53.145 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T12:06:53.147 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T12:06:53.151 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T12:06:53.152 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T12:06:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-16T12:06:53.258 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T12:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T12:06:53.260 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T12:06:53.261 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T12:06:53.262 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T12:06:53.263 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T12:06:53.264 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T12:06:53.265 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T12:06:53.266 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T12:06:53.267 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T12:06:53.269 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T12:06:53.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T12:06:53.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T12:06:53.272 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T12:06:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:Install 150 Packages 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 24 Packages 2024-03-16T12:06:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:06:53.275 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 224 M 2024-03-16T12:06:53.275 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-16T12:06:54.050 INFO:teuthology.orchestra.run.smithi098.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-16T12:06:54.409 INFO:teuthology.orchestra.run.smithi098.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.4 MB/s | 890 kB 00:00 2024-03-16T12:06:54.500 INFO:teuthology.orchestra.run.smithi098.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.6 MB/s | 145 kB 00:00 2024-03-16T12:06:54.984 INFO:teuthology.orchestra.run.smithi098.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 4.4 MB/s | 5.4 MB 00:01 2024-03-16T12:06:55.010 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:06:55.011 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:06:55.025 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T12:06:55.026 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T12:06:55.027 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T12:06:55.028 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T12:06:55.030 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T12:06:55.033 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T12:06:55.034 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T12:06:55.084 INFO:teuthology.orchestra.run.smithi098.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 4.1 MB/s | 2.4 MB 00:00 2024-03-16T12:06:55.114 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T12:06:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T12:06:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T12:06:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T12:06:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T12:06:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T12:06:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T12:06:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T12:06:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T12:06:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T12:06:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T12:06:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T12:06:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T12:06:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout:Install 150 Packages 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 24 Packages 2024-03-16T12:06:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:06:55.155 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 224 M 2024-03-16T12:06:55.155 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-16T12:06:55.226 INFO:teuthology.orchestra.run.smithi098.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-16T12:06:55.493 INFO:teuthology.orchestra.run.smithi098.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 12 MB/s | 21 MB 00:01 2024-03-16T12:06:55.805 INFO:teuthology.orchestra.run.smithi173.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-16T12:06:55.844 INFO:teuthology.orchestra.run.smithi098.stdout:(8/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 6.2 MB/s | 4.7 MB 00:00 2024-03-16T12:06:55.944 INFO:teuthology.orchestra.run.smithi098.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 254 kB/s | 25 kB 00:00 2024-03-16T12:06:56.511 INFO:teuthology.orchestra.run.smithi098.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 9.7 MB/s | 9.8 MB 00:01 2024-03-16T12:06:56.586 INFO:teuthology.orchestra.run.smithi098.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 221 kB 00:00 2024-03-16T12:06:56.676 INFO:teuthology.orchestra.run.smithi098.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 305 kB/s | 27 kB 00:00 2024-03-16T12:06:56.706 INFO:teuthology.orchestra.run.smithi173.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 989 kB/s | 890 kB 00:00 2024-03-16T12:06:56.728 INFO:teuthology.orchestra.run.smithi098.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 617 kB/s | 32 kB 00:00 2024-03-16T12:06:56.856 INFO:teuthology.orchestra.run.smithi173.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 966 kB/s | 145 kB 00:00 2024-03-16T12:06:56.970 INFO:teuthology.orchestra.run.smithi098.stdout:(14/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 9.7 MB/s | 17 MB 00:01 2024-03-16T12:06:57.020 INFO:teuthology.orchestra.run.smithi098.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.5 MB/s | 747 kB 00:00 2024-03-16T12:06:57.054 INFO:teuthology.orchestra.run.smithi098.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 163 kB 00:00 2024-03-16T12:06:57.088 INFO:teuthology.orchestra.run.smithi098.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-16T12:06:57.271 INFO:teuthology.orchestra.run.smithi098.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.3 MB/s | 501 kB 00:00 2024-03-16T12:06:57.372 INFO:teuthology.orchestra.run.smithi098.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 452 kB/s | 45 kB 00:00 2024-03-16T12:06:57.522 INFO:teuthology.orchestra.run.smithi098.stdout:(20/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-16T12:06:57.555 INFO:teuthology.orchestra.run.smithi098.stdout:(21/174): python3-ceph-common-19.0.0-2162.g3d78 718 kB/s | 131 kB 00:00 2024-03-16T12:06:57.589 INFO:teuthology.orchestra.run.smithi098.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.4 MB/s | 162 kB 00:00 2024-03-16T12:06:57.631 INFO:teuthology.orchestra.run.smithi098.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.2 MB/s | 320 kB 00:00 2024-03-16T12:06:57.665 INFO:teuthology.orchestra.run.smithi098.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 298 kB 00:00 2024-03-16T12:06:57.698 INFO:teuthology.orchestra.run.smithi098.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-16T12:06:57.716 INFO:teuthology.orchestra.run.smithi098.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-16T12:06:57.824 INFO:teuthology.orchestra.run.smithi098.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-16T12:06:57.908 INFO:teuthology.orchestra.run.smithi098.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 409 kB/s | 34 kB 00:00 2024-03-16T12:06:57.992 INFO:teuthology.orchestra.run.smithi098.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.6 MB/s | 139 kB 00:00 2024-03-16T12:06:58.084 INFO:teuthology.orchestra.run.smithi098.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 8.1 MB/s | 3.1 MB 00:00 2024-03-16T12:06:58.119 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-16T12:06:58.367 INFO:teuthology.orchestra.run.smithi098.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 4.8 MB/s | 1.8 MB 00:00 2024-03-16T12:06:58.459 INFO:teuthology.orchestra.run.smithi098.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.7 MB/s | 248 kB 00:00 2024-03-16T12:06:58.543 INFO:teuthology.orchestra.run.smithi098.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 594 kB/s | 49 kB 00:00 2024-03-16T12:06:58.610 INFO:teuthology.orchestra.run.smithi098.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 250 kB/s | 16 kB 00:00 2024-03-16T12:06:58.639 INFO:teuthology.orchestra.run.smithi173.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.3 MB/s | 2.4 MB 00:01 2024-03-16T12:06:58.710 INFO:teuthology.orchestra.run.smithi098.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.6 MB/s | 264 kB 00:00 2024-03-16T12:06:58.873 INFO:teuthology.orchestra.run.smithi173.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-03-16T12:06:59.044 INFO:teuthology.orchestra.run.smithi098.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 7.7 MB/s | 7.4 MB 00:00 2024-03-16T12:06:59.094 INFO:teuthology.orchestra.run.smithi098.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 1.9 MB/s | 748 kB 00:00 2024-03-16T12:06:59.119 INFO:teuthology.orchestra.run.smithi098.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 246 kB/s | 18 kB 00:00 2024-03-16T12:06:59.386 INFO:teuthology.orchestra.run.smithi098.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-16T12:06:59.453 INFO:teuthology.orchestra.run.smithi098.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 920 kB 00:00 2024-03-16T12:06:59.478 INFO:teuthology.orchestra.run.smithi098.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 106 kB 00:00 2024-03-16T12:06:59.498 INFO:teuthology.orchestra.run.smithi173.stdout:(6/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 5.1 MB/s | 21 MB 00:04 2024-03-16T12:06:59.512 INFO:teuthology.orchestra.run.smithi098.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-16T12:06:59.560 INFO:teuthology.orchestra.run.smithi098.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 403 kB/s | 32 kB 00:00 2024-03-16T12:06:59.704 INFO:teuthology.orchestra.run.smithi098.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-16T12:06:59.707 INFO:teuthology.orchestra.run.smithi173.stdout:(7/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-03-16T12:06:59.721 INFO:teuthology.orchestra.run.smithi098.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 111 kB/s | 17 kB 00:00 2024-03-16T12:06:59.746 INFO:teuthology.orchestra.run.smithi098.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 422 kB/s | 10 kB 00:00 2024-03-16T12:06:59.771 INFO:teuthology.orchestra.run.smithi098.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-16T12:06:59.797 INFO:teuthology.orchestra.run.smithi098.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 918 kB/s | 44 kB 00:00 2024-03-16T12:06:59.839 INFO:teuthology.orchestra.run.smithi098.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-16T12:06:59.872 INFO:teuthology.orchestra.run.smithi098.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.2 MB/s | 578 kB 00:00 2024-03-16T12:06:59.898 INFO:teuthology.orchestra.run.smithi098.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 865 kB/s | 46 kB 00:00 2024-03-16T12:06:59.932 INFO:teuthology.orchestra.run.smithi098.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-16T12:06:59.965 INFO:teuthology.orchestra.run.smithi098.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-16T12:07:00.007 INFO:teuthology.orchestra.run.smithi098.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2024-03-16T12:07:00.032 INFO:teuthology.orchestra.run.smithi098.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 514 kB/s | 34 kB 00:00 2024-03-16T12:07:00.141 INFO:teuthology.orchestra.run.smithi098.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-03-16T12:07:00.267 INFO:teuthology.orchestra.run.smithi098.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-16T12:07:00.308 INFO:teuthology.orchestra.run.smithi098.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 5.1 MB/s | 811 kB 00:00 2024-03-16T12:07:00.350 INFO:teuthology.orchestra.run.smithi098.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-16T12:07:00.549 INFO:teuthology.orchestra.run.smithi173.stdout:(8/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-16T12:07:00.551 INFO:teuthology.orchestra.run.smithi098.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2024-03-16T12:07:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-03-16T12:07:00.607 INFO:teuthology.orchestra.run.smithi173.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 433 kB/s | 25 kB 00:00 2024-03-16T12:07:00.626 INFO:teuthology.orchestra.run.smithi098.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 78 kB 00:00 2024-03-16T12:07:00.785 INFO:teuthology.orchestra.run.smithi098.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-16T12:07:00.810 INFO:teuthology.orchestra.run.smithi098.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 144 kB/s | 24 kB 00:00 2024-03-16T12:07:00.852 INFO:teuthology.orchestra.run.smithi098.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 448 kB/s | 18 kB 00:00 2024-03-16T12:07:00.878 INFO:teuthology.orchestra.run.smithi098.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-16T12:07:00.903 INFO:teuthology.orchestra.run.smithi098.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 652 kB/s | 32 kB 00:00 2024-03-16T12:07:00.999 INFO:teuthology.orchestra.run.smithi173.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 11 MB/s | 17 MB 00:01 2024-03-16T12:07:01.053 INFO:teuthology.orchestra.run.smithi098.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-16T12:07:01.066 INFO:teuthology.orchestra.run.smithi173.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.3 MB/s | 221 kB 00:00 2024-03-16T12:07:01.087 INFO:teuthology.orchestra.run.smithi098.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-16T12:07:01.112 INFO:teuthology.orchestra.run.smithi098.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.1 MB/s | 174 kB 00:00 2024-03-16T12:07:01.125 INFO:teuthology.orchestra.run.smithi173.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 463 kB/s | 27 kB 00:00 2024-03-16T12:07:01.137 INFO:teuthology.orchestra.run.smithi098.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-16T12:07:01.163 INFO:teuthology.orchestra.run.smithi098.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 41 kB 00:00 2024-03-16T12:07:01.184 INFO:teuthology.orchestra.run.smithi173.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 544 kB/s | 32 kB 00:00 2024-03-16T12:07:01.205 INFO:teuthology.orchestra.run.smithi098.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-16T12:07:01.230 INFO:teuthology.orchestra.run.smithi098.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-16T12:07:01.292 INFO:teuthology.orchestra.run.smithi173.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 6.8 MB/s | 747 kB 00:00 2024-03-16T12:07:01.295 INFO:teuthology.orchestra.run.smithi098.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-03-16T12:07:01.322 INFO:teuthology.orchestra.run.smithi098.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 529 kB/s | 47 kB 00:00 2024-03-16T12:07:01.339 INFO:teuthology.orchestra.run.smithi098.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-16T12:07:01.359 INFO:teuthology.orchestra.run.smithi173.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 163 kB 00:00 2024-03-16T12:07:01.426 INFO:teuthology.orchestra.run.smithi173.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-16T12:07:01.523 INFO:teuthology.orchestra.run.smithi098.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 219 kB/s | 39 kB 00:00 2024-03-16T12:07:01.526 INFO:teuthology.orchestra.run.smithi173.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.9 MB/s | 501 kB 00:00 2024-03-16T12:07:01.774 INFO:teuthology.orchestra.run.smithi098.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-16T12:07:01.791 INFO:teuthology.orchestra.run.smithi098.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 158 kB/s | 41 kB 00:00 2024-03-16T12:07:01.816 INFO:teuthology.orchestra.run.smithi098.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-03-16T12:07:01.865 INFO:teuthology.orchestra.run.smithi098.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 774 kB/s | 56 kB 00:00 2024-03-16T12:07:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 846 kB/s | 63 kB 00:00 2024-03-16T12:07:01.908 INFO:teuthology.orchestra.run.smithi098.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-16T12:07:01.934 INFO:teuthology.orchestra.run.smithi098.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-16T12:07:01.959 INFO:teuthology.orchestra.run.smithi098.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-03-16T12:07:01.985 INFO:teuthology.orchestra.run.smithi173.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-16T12:07:02.009 INFO:teuthology.orchestra.run.smithi098.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.1 MB 00:00 2024-03-16T12:07:02.034 INFO:teuthology.orchestra.run.smithi098.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-16T12:07:02.043 INFO:teuthology.orchestra.run.smithi173.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 778 kB/s | 45 kB 00:00 2024-03-16T12:07:02.060 INFO:teuthology.orchestra.run.smithi098.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-16T12:07:02.085 INFO:teuthology.orchestra.run.smithi098.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T12:07:02.110 INFO:teuthology.orchestra.run.smithi173.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.9 MB/s | 131 kB 00:00 2024-03-16T12:07:02.169 INFO:teuthology.orchestra.run.smithi173.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.7 MB/s | 162 kB 00:00 2024-03-16T12:07:02.244 INFO:teuthology.orchestra.run.smithi173.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.2 MB/s | 320 kB 00:00 2024-03-16T12:07:02.320 INFO:teuthology.orchestra.run.smithi173.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 298 kB 00:00 2024-03-16T12:07:02.387 INFO:teuthology.orchestra.run.smithi173.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-16T12:07:02.446 INFO:teuthology.orchestra.run.smithi173.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-16T12:07:02.746 INFO:teuthology.orchestra.run.smithi173.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 10 MB/s | 3.1 MB 00:00 2024-03-16T12:07:02.813 INFO:teuthology.orchestra.run.smithi173.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-16T12:07:02.871 INFO:teuthology.orchestra.run.smithi173.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 584 kB/s | 34 kB 00:00 2024-03-16T12:07:02.963 INFO:teuthology.orchestra.run.smithi173.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-16T12:07:02.974 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-16T12:07:03.145 INFO:teuthology.orchestra.run.smithi098.stdout:(91/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 6.6 MB/s | 48 MB 00:07 2024-03-16T12:07:03.205 INFO:teuthology.orchestra.run.smithi173.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 7.4 MB/s | 1.8 MB 00:00 2024-03-16T12:07:03.503 INFO:teuthology.orchestra.run.smithi098.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 130 kB/s | 187 kB 00:01 2024-03-16T12:07:03.681 INFO:teuthology.orchestra.run.smithi173.stdout:(31/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 2.5 MB/s | 9.8 MB 00:03 2024-03-16T12:07:03.831 INFO:teuthology.orchestra.run.smithi173.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.6 MB/s | 248 kB 00:00 2024-03-16T12:07:03.973 INFO:teuthology.orchestra.run.smithi173.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 9.7 MB/s | 7.4 MB 00:00 2024-03-16T12:07:03.998 INFO:teuthology.orchestra.run.smithi173.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 296 kB/s | 49 kB 00:00 2024-03-16T12:07:04.032 INFO:teuthology.orchestra.run.smithi173.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 281 kB/s | 16 kB 00:00 2024-03-16T12:07:04.115 INFO:teuthology.orchestra.run.smithi173.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.2 MB/s | 264 kB 00:00 2024-03-16T12:07:04.120 INFO:teuthology.orchestra.run.smithi098.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 201 kB/s | 195 kB 00:00 2024-03-16T12:07:04.149 INFO:teuthology.orchestra.run.smithi173.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 6.2 MB/s | 748 kB 00:00 2024-03-16T12:07:04.174 INFO:teuthology.orchestra.run.smithi173.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 311 kB/s | 18 kB 00:00 2024-03-16T12:07:04.212 INFO:teuthology.orchestra.run.smithi098.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 239 kB/s | 21 kB 00:00 2024-03-16T12:07:04.254 INFO:teuthology.orchestra.run.smithi098.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 621 kB/s | 25 kB 00:00 2024-03-16T12:07:04.333 INFO:teuthology.orchestra.run.smithi173.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-16T12:07:04.366 INFO:teuthology.orchestra.run.smithi173.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2024-03-16T12:07:04.392 INFO:teuthology.orchestra.run.smithi173.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-16T12:07:04.412 INFO:teuthology.orchestra.run.smithi098.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 137 kB/s | 125 kB 00:00 2024-03-16T12:07:04.417 INFO:teuthology.orchestra.run.smithi173.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T12:07:04.434 INFO:teuthology.orchestra.run.smithi173.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 763 kB/s | 32 kB 00:00 2024-03-16T12:07:04.510 INFO:teuthology.orchestra.run.smithi173.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 228 kB/s | 17 kB 00:00 2024-03-16T12:07:04.521 INFO:teuthology.orchestra.run.smithi098.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 330 kB/s | 803 kB 00:02 2024-03-16T12:07:04.602 INFO:teuthology.orchestra.run.smithi173.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-16T12:07:04.618 INFO:teuthology.orchestra.run.smithi173.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 94 kB/s | 10 kB 00:00 2024-03-16T12:07:04.629 INFO:teuthology.orchestra.run.smithi098.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-03-16T12:07:04.654 INFO:teuthology.orchestra.run.smithi098.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-16T12:07:04.660 INFO:teuthology.orchestra.run.smithi173.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-16T12:07:04.677 INFO:teuthology.orchestra.run.smithi173.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-16T12:07:04.688 INFO:teuthology.orchestra.run.smithi098.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.7 MB/s | 216 kB 00:00 2024-03-16T12:07:04.719 INFO:teuthology.orchestra.run.smithi173.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-16T12:07:04.731 INFO:teuthology.orchestra.run.smithi098.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 459 kB/s | 35 kB 00:00 2024-03-16T12:07:04.745 INFO:teuthology.orchestra.run.smithi173.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-16T12:07:04.755 INFO:teuthology.orchestra.run.smithi098.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 673 kB/s | 44 kB 00:00 2024-03-16T12:07:04.762 INFO:teuthology.orchestra.run.smithi173.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-16T12:07:04.781 INFO:teuthology.orchestra.run.smithi098.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-03-16T12:07:04.795 INFO:teuthology.orchestra.run.smithi173.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T12:07:04.814 INFO:teuthology.orchestra.run.smithi098.stdout:(104/174): python3-jinja2-2.11.3-5.el9.noarch.r 849 kB/s | 248 kB 00:00 2024-03-16T12:07:04.821 INFO:teuthology.orchestra.run.smithi173.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-16T12:07:04.846 INFO:teuthology.orchestra.run.smithi173.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-16T12:07:04.863 INFO:teuthology.orchestra.run.smithi173.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-16T12:07:04.997 INFO:teuthology.orchestra.run.smithi173.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-16T12:07:05.030 INFO:teuthology.orchestra.run.smithi173.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 9.1 MB/s | 1.5 MB 00:00 2024-03-16T12:07:05.064 INFO:teuthology.orchestra.run.smithi173.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-16T12:07:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 69 kB/s | 22 kB 00:00 2024-03-16T12:07:05.089 INFO:teuthology.orchestra.run.smithi173.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-16T12:07:05.240 INFO:teuthology.orchestra.run.smithi173.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-16T12:07:05.282 INFO:teuthology.orchestra.run.smithi173.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-16T12:07:05.307 INFO:teuthology.orchestra.run.smithi173.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-16T12:07:05.332 INFO:teuthology.orchestra.run.smithi098.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-03-16T12:07:05.413 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:07:05.414 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T12:07:05.429 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T12:07:05.430 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T12:07:05.431 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T12:07:05.432 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T12:07:05.434 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T12:07:05.437 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T12:07:05.438 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T12:07:05.466 INFO:teuthology.orchestra.run.smithi173.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 5.8 MB 00:00 2024-03-16T12:07:05.491 INFO:teuthology.orchestra.run.smithi173.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 133 kB/s | 24 kB 00:00 2024-03-16T12:07:05.517 INFO:teuthology.orchestra.run.smithi173.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 359 kB/s | 18 kB 00:00 2024-03-16T12:07:05.523 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-16T12:07:05.542 INFO:teuthology.orchestra.run.smithi173.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-16T12:07:05.547 INFO:teuthology.orchestra.run.smithi098.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 750 kB/s | 548 kB 00:00 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T12:07:05.553 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T12:07:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T12:07:05.555 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T12:07:05.556 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T12:07:05.557 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T12:07:05.558 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T12:07:05.559 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T12:07:05.560 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T12:07:05.561 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T12:07:05.562 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T12:07:05.563 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T12:07:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout:Install 150 Packages 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 24 Packages 2024-03-16T12:07:05.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:05.566 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 224 M 2024-03-16T12:07:05.566 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-16T12:07:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-16T12:07:05.665 INFO:teuthology.orchestra.run.smithi098.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 532 kB/s | 309 kB 00:00 2024-03-16T12:07:05.708 INFO:teuthology.orchestra.run.smithi098.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-03-16T12:07:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-16T12:07:05.743 INFO:teuthology.orchestra.run.smithi173.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-16T12:07:05.760 INFO:teuthology.orchestra.run.smithi173.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-16T12:07:05.766 INFO:teuthology.orchestra.run.smithi098.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-16T12:07:05.819 INFO:teuthology.orchestra.run.smithi173.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 913 kB/s | 69 kB 00:00 2024-03-16T12:07:05.835 INFO:teuthology.orchestra.run.smithi173.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 546 kB/s | 41 kB 00:00 2024-03-16T12:07:05.869 INFO:teuthology.orchestra.run.smithi173.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T12:07:05.894 INFO:teuthology.orchestra.run.smithi173.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-16T12:07:05.898 INFO:teuthology.orchestra.run.smithi098.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 516 kB/s | 67 kB 00:00 2024-03-16T12:07:05.920 INFO:teuthology.orchestra.run.smithi173.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-16T12:07:05.933 INFO:teuthology.orchestra.run.smithi098.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 216 kB/s | 49 kB 00:00 2024-03-16T12:07:05.937 INFO:teuthology.orchestra.run.smithi173.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-16T12:07:05.954 INFO:teuthology.orchestra.run.smithi173.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-16T12:07:06.051 INFO:teuthology.orchestra.run.smithi098.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-03-16T12:07:06.084 INFO:teuthology.orchestra.run.smithi098.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 815 kB/s | 151 kB 00:00 2024-03-16T12:07:06.121 INFO:teuthology.orchestra.run.smithi173.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 232 kB/s | 39 kB 00:00 2024-03-16T12:07:06.159 INFO:teuthology.orchestra.run.smithi098.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-03-16T12:07:06.177 INFO:teuthology.orchestra.run.smithi098.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 475 kB/s | 43 kB 00:00 2024-03-16T12:07:06.235 INFO:teuthology.orchestra.run.smithi098.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-03-16T12:07:06.260 INFO:teuthology.orchestra.run.smithi098.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 181 kB/s | 15 kB 00:00 2024-03-16T12:07:06.292 INFO:teuthology.orchestra.run.smithi153.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-03-16T12:07:06.302 INFO:teuthology.orchestra.run.smithi098.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-16T12:07:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-16T12:07:06.386 INFO:teuthology.orchestra.run.smithi098.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-16T12:07:06.403 INFO:teuthology.orchestra.run.smithi098.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 532 kB/s | 35 kB 00:00 2024-03-16T12:07:06.404 INFO:teuthology.orchestra.run.smithi173.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-16T12:07:06.421 INFO:teuthology.orchestra.run.smithi173.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-16T12:07:06.447 INFO:teuthology.orchestra.run.smithi173.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-16T12:07:06.464 INFO:teuthology.orchestra.run.smithi173.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-16T12:07:06.467 INFO:teuthology.orchestra.run.smithi098.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-03-16T12:07:06.481 INFO:teuthology.orchestra.run.smithi173.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-16T12:07:06.498 INFO:teuthology.orchestra.run.smithi173.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-16T12:07:06.512 INFO:teuthology.orchestra.run.smithi098.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-16T12:07:06.523 INFO:teuthology.orchestra.run.smithi173.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-16T12:07:06.546 INFO:teuthology.orchestra.run.smithi098.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.8 MB/s | 144 kB 00:00 2024-03-16T12:07:06.557 INFO:teuthology.orchestra.run.smithi173.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-16T12:07:06.588 INFO:teuthology.orchestra.run.smithi098.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-16T12:07:06.590 INFO:teuthology.orchestra.run.smithi173.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-16T12:07:06.616 INFO:teuthology.orchestra.run.smithi173.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-16T12:07:06.621 INFO:teuthology.orchestra.run.smithi098.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 237 kB/s | 18 kB 00:00 2024-03-16T12:07:06.641 INFO:teuthology.orchestra.run.smithi173.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-16T12:07:06.658 INFO:teuthology.orchestra.run.smithi173.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-16T12:07:06.680 INFO:teuthology.orchestra.run.smithi098.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-03-16T12:07:06.700 INFO:teuthology.orchestra.run.smithi153.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.9 MB/s | 5.4 MB 00:00 2024-03-16T12:07:06.705 INFO:teuthology.orchestra.run.smithi098.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-16T12:07:06.722 INFO:teuthology.orchestra.run.smithi098.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-03-16T12:07:06.747 INFO:teuthology.orchestra.run.smithi098.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 297 kB/s | 20 kB 00:00 2024-03-16T12:07:06.776 INFO:teuthology.orchestra.run.smithi153.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.9 MB/s | 145 kB 00:00 2024-03-16T12:07:06.798 INFO:teuthology.orchestra.run.smithi098.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 624 kB/s | 46 kB 00:00 2024-03-16T12:07:06.800 INFO:teuthology.orchestra.run.smithi173.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-16T12:07:06.823 INFO:teuthology.orchestra.run.smithi098.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-16T12:07:06.859 INFO:teuthology.orchestra.run.smithi173.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-03-16T12:07:06.873 INFO:teuthology.orchestra.run.smithi098.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 776 kB/s | 58 kB 00:00 2024-03-16T12:07:06.884 INFO:teuthology.orchestra.run.smithi173.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-16T12:07:06.909 INFO:teuthology.orchestra.run.smithi173.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-16T12:07:06.915 INFO:teuthology.orchestra.run.smithi098.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-16T12:07:06.957 INFO:teuthology.orchestra.run.smithi098.stdout:(136/174): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-03-16T12:07:06.983 INFO:teuthology.orchestra.run.smithi098.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-16T12:07:07.008 INFO:teuthology.orchestra.run.smithi098.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 982 kB/s | 90 kB 00:00 2024-03-16T12:07:07.058 INFO:teuthology.orchestra.run.smithi098.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-03-16T12:07:07.084 INFO:teuthology.orchestra.run.smithi153.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-03-16T12:07:07.088 INFO:teuthology.orchestra.run.smithi098.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 294 kB/s | 31 kB 00:00 2024-03-16T12:07:07.122 INFO:teuthology.orchestra.run.smithi098.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-16T12:07:07.147 INFO:teuthology.orchestra.run.smithi098.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 675 kB/s | 59 kB 00:00 2024-03-16T12:07:07.172 INFO:teuthology.orchestra.run.smithi098.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 429 kB/s | 36 kB 00:00 2024-03-16T12:07:07.198 INFO:teuthology.orchestra.run.smithi098.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-03-16T12:07:07.223 INFO:teuthology.orchestra.run.smithi098.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-16T12:07:07.226 INFO:teuthology.orchestra.run.smithi153.stdout:(5/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 953 kB/s | 890 kB 00:00 2024-03-16T12:07:07.257 INFO:teuthology.orchestra.run.smithi098.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-16T12:07:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-03-16T12:07:07.301 INFO:teuthology.orchestra.run.smithi153.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-03-16T12:07:07.307 INFO:teuthology.orchestra.run.smithi098.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-16T12:07:07.341 INFO:teuthology.orchestra.run.smithi098.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-16T12:07:07.416 INFO:teuthology.orchestra.run.smithi098.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-16T12:07:07.476 INFO:teuthology.orchestra.run.smithi173.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-03-16T12:07:07.518 INFO:teuthology.orchestra.run.smithi173.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 42 kB/s | 25 kB 00:00 2024-03-16T12:07:07.642 INFO:teuthology.orchestra.run.smithi098.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.0 MB/s | 321 kB 00:00 2024-03-16T12:07:07.668 INFO:teuthology.orchestra.run.smithi153.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 13 MB/s | 21 MB 00:01 2024-03-16T12:07:08.250 INFO:teuthology.orchestra.run.smithi098.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-16T12:07:08.427 INFO:teuthology.orchestra.run.smithi173.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 270 kB/s | 245 kB 00:00 2024-03-16T12:07:08.502 INFO:teuthology.orchestra.run.smithi153.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 12 MB/s | 9.8 MB 00:00 2024-03-16T12:07:08.544 INFO:teuthology.orchestra.run.smithi153.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 608 kB/s | 25 kB 00:00 2024-03-16T12:07:09.327 INFO:teuthology.orchestra.run.smithi173.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 276 kB/s | 248 kB 00:00 2024-03-16T12:07:09.434 INFO:teuthology.orchestra.run.smithi098.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.5 MB/s | 3.1 MB 00:02 2024-03-16T12:07:09.452 INFO:teuthology.orchestra.run.smithi153.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 7.8 MB/s | 17 MB 00:02 2024-03-16T12:07:09.502 INFO:teuthology.orchestra.run.smithi153.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 4.4 MB/s | 221 kB 00:00 2024-03-16T12:07:09.561 INFO:teuthology.orchestra.run.smithi153.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 465 kB/s | 27 kB 00:00 2024-03-16T12:07:09.620 INFO:teuthology.orchestra.run.smithi153.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 544 kB/s | 32 kB 00:00 2024-03-16T12:07:09.762 INFO:teuthology.orchestra.run.smithi153.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 5.1 MB/s | 747 kB 00:00 2024-03-16T12:07:09.837 INFO:teuthology.orchestra.run.smithi153.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-16T12:07:09.912 INFO:teuthology.orchestra.run.smithi153.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-16T12:07:10.004 INFO:teuthology.orchestra.run.smithi153.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 5.3 MB/s | 501 kB 00:00 2024-03-16T12:07:10.142 INFO:teuthology.orchestra.run.smithi098.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 60 kB/s | 42 kB 00:00 2024-03-16T12:07:10.260 INFO:teuthology.orchestra.run.smithi173.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 232 kB/s | 216 kB 00:00 2024-03-16T12:07:10.629 INFO:teuthology.orchestra.run.smithi153.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 8.2 MB/s | 5.1 MB 00:00 2024-03-16T12:07:10.697 INFO:teuthology.orchestra.run.smithi153.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 675 kB/s | 45 kB 00:00 2024-03-16T12:07:10.780 INFO:teuthology.orchestra.run.smithi153.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T12:07:10.855 INFO:teuthology.orchestra.run.smithi153.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 162 kB 00:00 2024-03-16T12:07:10.861 INFO:teuthology.orchestra.run.smithi173.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 58 kB/s | 35 kB 00:00 2024-03-16T12:07:10.939 INFO:teuthology.orchestra.run.smithi153.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 320 kB 00:00 2024-03-16T12:07:11.023 INFO:teuthology.orchestra.run.smithi153.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-16T12:07:11.106 INFO:teuthology.orchestra.run.smithi153.stdout:(24/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 4.7 MB 00:03 2024-03-16T12:07:11.132 INFO:teuthology.orchestra.run.smithi153.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 915 kB/s | 99 kB 00:00 2024-03-16T12:07:11.134 INFO:teuthology.orchestra.run.smithi098.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 251 kB/s | 248 kB 00:00 2024-03-16T12:07:11.301 INFO:teuthology.orchestra.run.smithi098.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 610 kB/s | 1.8 MB 00:03 2024-03-16T12:07:11.360 INFO:teuthology.orchestra.run.smithi098.stdout:(157/174): NetworkManager-1.46.0-1.el9.x86_64.r 631 kB/s | 2.3 MB 00:03 2024-03-16T12:07:11.486 INFO:teuthology.orchestra.run.smithi173.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-16T12:07:11.615 INFO:teuthology.orchestra.run.smithi153.stdout:(26/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-16T12:07:11.640 INFO:teuthology.orchestra.run.smithi153.stdout:(27/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 160 kB/s | 85 kB 00:00 2024-03-16T12:07:11.666 INFO:teuthology.orchestra.run.smithi153.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-16T12:07:11.699 INFO:teuthology.orchestra.run.smithi153.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 580 kB/s | 34 kB 00:00 2024-03-16T12:07:11.758 INFO:teuthology.orchestra.run.smithi153.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.9 MB/s | 3.1 MB 00:00 2024-03-16T12:07:11.783 INFO:teuthology.orchestra.run.smithi153.stdout:(31/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.2 MB/s | 139 kB 00:00 2024-03-16T12:07:11.842 INFO:teuthology.orchestra.run.smithi153.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.2 MB/s | 248 kB 00:00 2024-03-16T12:07:11.892 INFO:teuthology.orchestra.run.smithi153.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 990 kB/s | 49 kB 00:00 2024-03-16T12:07:11.942 INFO:teuthology.orchestra.run.smithi153.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 331 kB/s | 16 kB 00:00 2024-03-16T12:07:12.009 INFO:teuthology.orchestra.run.smithi153.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 264 kB 00:00 2024-03-16T12:07:12.019 INFO:teuthology.orchestra.run.smithi098.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 51 kB/s | 36 kB 00:00 2024-03-16T12:07:12.045 INFO:teuthology.orchestra.run.smithi098.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 79 kB/s | 71 kB 00:00 2024-03-16T12:07:12.135 INFO:teuthology.orchestra.run.smithi153.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 5.9 MB/s | 748 kB 00:00 2024-03-16T12:07:12.168 INFO:teuthology.orchestra.run.smithi153.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 548 kB/s | 18 kB 00:00 2024-03-16T12:07:12.253 INFO:teuthology.orchestra.run.smithi173.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 122 kB/s | 94 kB 00:00 2024-03-16T12:07:12.260 INFO:teuthology.orchestra.run.smithi153.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-16T12:07:12.278 INFO:teuthology.orchestra.run.smithi098.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 210 kB/s | 192 kB 00:00 2024-03-16T12:07:12.395 INFO:teuthology.orchestra.run.smithi173.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 158 kB/s | 22 kB 00:00 2024-03-16T12:07:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 662 kB 00:00 2024-03-16T12:07:12.527 INFO:teuthology.orchestra.run.smithi153.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-16T12:07:12.553 INFO:teuthology.orchestra.run.smithi153.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-16T12:07:12.578 INFO:teuthology.orchestra.run.smithi153.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T12:07:12.595 INFO:teuthology.orchestra.run.smithi153.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 768 kB/s | 32 kB 00:00 2024-03-16T12:07:12.670 INFO:teuthology.orchestra.run.smithi153.stdout:(44/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 1.8 MB/s | 1.8 MB 00:00 2024-03-16T12:07:12.745 INFO:teuthology.orchestra.run.smithi153.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-16T12:07:12.762 INFO:teuthology.orchestra.run.smithi153.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 103 kB/s | 17 kB 00:00 2024-03-16T12:07:12.779 INFO:teuthology.orchestra.run.smithi153.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 94 kB/s | 10 kB 00:00 2024-03-16T12:07:12.799 INFO:teuthology.orchestra.run.smithi098.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 31 kB/s | 23 kB 00:00 2024-03-16T12:07:12.804 INFO:teuthology.orchestra.run.smithi153.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-16T12:07:12.830 INFO:teuthology.orchestra.run.smithi153.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-16T12:07:12.864 INFO:teuthology.orchestra.run.smithi153.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-16T12:07:12.897 INFO:teuthology.orchestra.run.smithi153.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-16T12:07:12.922 INFO:teuthology.orchestra.run.smithi153.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 511 kB/s | 46 kB 00:00 2024-03-16T12:07:12.956 INFO:teuthology.orchestra.run.smithi153.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-16T12:07:12.981 INFO:teuthology.orchestra.run.smithi153.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-16T12:07:13.006 INFO:teuthology.orchestra.run.smithi153.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-03-16T12:07:13.031 INFO:teuthology.orchestra.run.smithi153.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 457 kB/s | 34 kB 00:00 2024-03-16T12:07:13.107 INFO:teuthology.orchestra.run.smithi153.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-16T12:07:13.224 INFO:teuthology.orchestra.run.smithi153.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-16T12:07:13.253 INFO:teuthology.orchestra.run.smithi098.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 207 kB/s | 256 kB 00:01 2024-03-16T12:07:13.265 INFO:teuthology.orchestra.run.smithi153.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.4 MB/s | 811 kB 00:00 2024-03-16T12:07:13.291 INFO:teuthology.orchestra.run.smithi153.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 741 kB/s | 134 kB 00:00 2024-03-16T12:07:13.428 INFO:teuthology.orchestra.run.smithi173.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 835 kB/s | 863 kB 00:01 2024-03-16T12:07:13.433 INFO:teuthology.orchestra.run.smithi153.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-16T12:07:13.437 INFO:teuthology.orchestra.run.smithi098.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 104 kB/s | 108 kB 00:01 2024-03-16T12:07:13.456 INFO:teuthology.orchestra.run.smithi153.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 479 kB/s | 78 kB 00:00 2024-03-16T12:07:13.490 INFO:teuthology.orchestra.run.smithi153.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-16T12:07:13.640 INFO:teuthology.orchestra.run.smithi153.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-16T12:07:13.665 INFO:teuthology.orchestra.run.smithi153.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 117 kB/s | 24 kB 00:00 2024-03-16T12:07:13.687 INFO:teuthology.orchestra.run.smithi173.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 681 kB/s | 4.1 MB 00:06 2024-03-16T12:07:13.691 INFO:teuthology.orchestra.run.smithi153.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 91 kB/s | 18 kB 00:00 2024-03-16T12:07:13.716 INFO:teuthology.orchestra.run.smithi153.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-16T12:07:13.750 INFO:teuthology.orchestra.run.smithi153.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 539 kB/s | 32 kB 00:00 2024-03-16T12:07:13.859 INFO:teuthology.orchestra.run.smithi153.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-16T12:07:13.879 INFO:teuthology.orchestra.run.smithi098.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 144 kB/s | 155 kB 00:01 2024-03-16T12:07:13.884 INFO:teuthology.orchestra.run.smithi153.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-16T12:07:13.909 INFO:teuthology.orchestra.run.smithi153.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-16T12:07:13.934 INFO:teuthology.orchestra.run.smithi153.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-16T12:07:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 994 kB/s | 41 kB 00:00 2024-03-16T12:07:13.976 INFO:teuthology.orchestra.run.smithi153.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-16T12:07:14.002 INFO:teuthology.orchestra.run.smithi153.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-16T12:07:14.027 INFO:teuthology.orchestra.run.smithi153.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-16T12:07:14.044 INFO:teuthology.orchestra.run.smithi153.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 712 kB/s | 47 kB 00:00 2024-03-16T12:07:14.061 INFO:teuthology.orchestra.run.smithi153.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 732 kB/s | 43 kB 00:00 2024-03-16T12:07:14.078 INFO:teuthology.orchestra.run.smithi153.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-16T12:07:14.146 INFO:teuthology.orchestra.run.smithi173.stdout:(106/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 3.5 MB/s | 48 MB 00:13 2024-03-16T12:07:14.211 INFO:teuthology.orchestra.run.smithi153.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 273 kB/s | 41 kB 00:00 2024-03-16T12:07:14.254 INFO:teuthology.orchestra.run.smithi098.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 136 kB/s | 136 kB 00:00 2024-03-16T12:07:14.270 INFO:teuthology.orchestra.run.smithi153.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-16T12:07:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 74 kB/s | 64 kB 00:00 2024-03-16T12:07:14.478 INFO:teuthology.orchestra.run.smithi153.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-16T12:07:14.495 INFO:teuthology.orchestra.run.smithi153.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 200 kB/s | 56 kB 00:00 2024-03-16T12:07:14.520 INFO:teuthology.orchestra.run.smithi153.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 252 kB/s | 63 kB 00:00 2024-03-16T12:07:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-16T12:07:14.562 INFO:teuthology.orchestra.run.smithi173.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-03-16T12:07:14.563 INFO:teuthology.orchestra.run.smithi153.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-16T12:07:14.588 INFO:teuthology.orchestra.run.smithi173.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 473 kB/s | 548 kB 00:01 2024-03-16T12:07:14.588 INFO:teuthology.orchestra.run.smithi153.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-16T12:07:14.622 INFO:teuthology.orchestra.run.smithi153.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T12:07:14.646 INFO:teuthology.orchestra.run.smithi098.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 37 kB/s | 29 kB 00:00 2024-03-16T12:07:14.647 INFO:teuthology.orchestra.run.smithi153.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-16T12:07:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-16T12:07:14.688 INFO:teuthology.orchestra.run.smithi173.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-16T12:07:14.697 INFO:teuthology.orchestra.run.smithi153.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T12:07:14.763 INFO:teuthology.orchestra.run.smithi173.stdout:(110/174): gperftools-libs-2.9.1-2.el9.x86_64.r 287 kB/s | 309 kB 00:01 2024-03-16T12:07:14.803 INFO:teuthology.orchestra.run.smithi173.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 424 kB/s | 49 kB 00:00 2024-03-16T12:07:14.822 INFO:teuthology.orchestra.run.smithi153.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-16T12:07:14.847 INFO:teuthology.orchestra.run.smithi153.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-16T12:07:14.881 INFO:teuthology.orchestra.run.smithi153.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-16T12:07:14.886 INFO:teuthology.orchestra.run.smithi173.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 547 kB/s | 67 kB 00:00 2024-03-16T12:07:14.907 INFO:teuthology.orchestra.run.smithi153.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-16T12:07:14.937 INFO:teuthology.orchestra.run.smithi173.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-16T12:07:14.957 INFO:teuthology.orchestra.run.smithi153.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-03-16T12:07:14.991 INFO:teuthology.orchestra.run.smithi153.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-03-16T12:07:15.074 INFO:teuthology.orchestra.run.smithi153.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-03-16T12:07:15.099 INFO:teuthology.orchestra.run.smithi153.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-16T12:07:15.125 INFO:teuthology.orchestra.run.smithi153.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-16T12:07:15.129 INFO:teuthology.orchestra.run.smithi173.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-16T12:07:15.150 INFO:teuthology.orchestra.run.smithi153.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 693 kB/s | 35 kB 00:00 2024-03-16T12:07:15.171 INFO:teuthology.orchestra.run.smithi173.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-03-16T12:07:15.175 INFO:teuthology.orchestra.run.smithi153.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-16T12:07:15.251 INFO:teuthology.orchestra.run.smithi153.stdout:(103/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-03-16T12:07:15.254 INFO:teuthology.orchestra.run.smithi173.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-03-16T12:07:15.288 INFO:teuthology.orchestra.run.smithi173.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-16T12:07:15.351 INFO:teuthology.orchestra.run.smithi153.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-16T12:07:15.355 INFO:teuthology.orchestra.run.smithi173.stdout:(118/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-03-16T12:07:15.372 INFO:teuthology.orchestra.run.smithi173.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-16T12:07:15.406 INFO:teuthology.orchestra.run.smithi173.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-16T12:07:15.451 INFO:teuthology.orchestra.run.smithi153.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-03-16T12:07:15.476 INFO:teuthology.orchestra.run.smithi153.stdout:(106/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 287 kB/s | 94 kB 00:00 2024-03-16T12:07:15.489 INFO:teuthology.orchestra.run.smithi173.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-16T12:07:15.548 INFO:teuthology.orchestra.run.smithi153.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-03-16T12:07:15.552 INFO:teuthology.orchestra.run.smithi153.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-16T12:07:15.594 INFO:teuthology.orchestra.run.smithi153.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-03-16T12:07:15.619 INFO:teuthology.orchestra.run.smithi153.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 387 kB/s | 26 kB 00:00 2024-03-16T12:07:15.644 INFO:teuthology.orchestra.run.smithi153.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-03-16T12:07:15.670 INFO:teuthology.orchestra.run.smithi153.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-16T12:07:15.673 INFO:teuthology.orchestra.run.smithi173.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 132 kB/s | 35 kB 00:00 2024-03-16T12:07:15.698 INFO:teuthology.orchestra.run.smithi173.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 1.0 MB/s | 358 kB 00:00 2024-03-16T12:07:15.722 INFO:teuthology.orchestra.run.smithi153.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.9 MB/s | 151 kB 00:00 2024-03-16T12:07:15.726 INFO:teuthology.orchestra.run.smithi173.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 857 kB/s | 203 kB 00:00 2024-03-16T12:07:15.796 INFO:teuthology.orchestra.run.smithi153.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-16T12:07:15.829 INFO:teuthology.orchestra.run.smithi153.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-03-16T12:07:15.834 INFO:teuthology.orchestra.run.smithi173.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-16T12:07:15.854 INFO:teuthology.orchestra.run.smithi153.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 745 kB/s | 43 kB 00:00 2024-03-16T12:07:15.859 INFO:teuthology.orchestra.run.smithi173.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-16T12:07:15.879 INFO:teuthology.orchestra.run.smithi153.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-03-16T12:07:15.905 INFO:teuthology.orchestra.run.smithi153.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-16T12:07:15.932 INFO:teuthology.orchestra.run.smithi153.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-16T12:07:15.943 INFO:teuthology.orchestra.run.smithi173.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-16T12:07:15.957 INFO:teuthology.orchestra.run.smithi153.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T12:07:15.977 INFO:teuthology.orchestra.run.smithi173.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-16T12:07:15.991 INFO:teuthology.orchestra.run.smithi153.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.1 MB/s | 358 kB 00:00 2024-03-16T12:07:16.016 INFO:teuthology.orchestra.run.smithi153.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-03-16T12:07:16.027 INFO:teuthology.orchestra.run.smithi173.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.8 MB/s | 2.0 MB 00:00 2024-03-16T12:07:16.041 INFO:teuthology.orchestra.run.smithi153.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-16T12:07:16.044 INFO:teuthology.orchestra.run.smithi173.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-03-16T12:07:16.067 INFO:teuthology.orchestra.run.smithi153.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-16T12:07:16.086 INFO:teuthology.orchestra.run.smithi173.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-16T12:07:16.092 INFO:teuthology.orchestra.run.smithi153.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-16T12:07:16.117 INFO:teuthology.orchestra.run.smithi153.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-16T12:07:16.142 INFO:teuthology.orchestra.run.smithi153.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-16T12:07:16.168 INFO:teuthology.orchestra.run.smithi153.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-03-16T12:07:16.169 INFO:teuthology.orchestra.run.smithi173.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-03-16T12:07:16.193 INFO:teuthology.orchestra.run.smithi153.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-03-16T12:07:16.203 INFO:teuthology.orchestra.run.smithi173.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-16T12:07:16.218 INFO:teuthology.orchestra.run.smithi153.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-16T12:07:16.228 INFO:teuthology.orchestra.run.smithi173.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 555 kB/s | 79 kB 00:00 2024-03-16T12:07:16.277 INFO:teuthology.orchestra.run.smithi153.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-03-16T12:07:16.279 INFO:teuthology.orchestra.run.smithi173.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-03-16T12:07:16.302 INFO:teuthology.orchestra.run.smithi153.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-03-16T12:07:16.319 INFO:teuthology.orchestra.run.smithi153.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-16T12:07:16.337 INFO:teuthology.orchestra.run.smithi173.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-16T12:07:16.353 INFO:teuthology.orchestra.run.smithi153.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-16T12:07:16.354 INFO:teuthology.orchestra.run.smithi173.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-16T12:07:16.370 INFO:teuthology.orchestra.run.smithi153.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 862 kB/s | 58 kB 00:00 2024-03-16T12:07:16.387 INFO:teuthology.orchestra.run.smithi153.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 490 kB/s | 16 kB 00:00 2024-03-16T12:07:16.388 INFO:teuthology.orchestra.run.smithi173.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-03-16T12:07:16.412 INFO:teuthology.orchestra.run.smithi153.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-16T12:07:16.429 INFO:teuthology.orchestra.run.smithi153.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 679 kB/s | 28 kB 00:00 2024-03-16T12:07:16.438 INFO:teuthology.orchestra.run.smithi173.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-16T12:07:16.446 INFO:teuthology.orchestra.run.smithi153.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 920 kB/s | 31 kB 00:00 2024-03-16T12:07:16.463 INFO:teuthology.orchestra.run.smithi173.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-03-16T12:07:16.471 INFO:teuthology.orchestra.run.smithi153.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-16T12:07:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-16T12:07:16.513 INFO:teuthology.orchestra.run.smithi173.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-16T12:07:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 547 kB/s | 1.2 MB 00:02 2024-03-16T12:07:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-16T12:07:16.547 INFO:teuthology.orchestra.run.smithi153.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-16T12:07:16.572 INFO:teuthology.orchestra.run.smithi173.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-03-16T12:07:16.572 INFO:teuthology.orchestra.run.smithi153.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-16T12:07:16.589 INFO:teuthology.orchestra.run.smithi173.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-03-16T12:07:16.598 INFO:teuthology.orchestra.run.smithi153.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-16T12:07:16.623 INFO:teuthology.orchestra.run.smithi153.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-16T12:07:16.631 INFO:teuthology.orchestra.run.smithi173.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 435 kB/s | 51 kB 00:00 2024-03-16T12:07:16.648 INFO:teuthology.orchestra.run.smithi153.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-03-16T12:07:16.673 INFO:teuthology.orchestra.run.smithi153.stdout:(148/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-16T12:07:16.698 INFO:teuthology.orchestra.run.smithi153.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T12:07:16.706 INFO:teuthology.orchestra.run.smithi173.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-16T12:07:16.723 INFO:teuthology.orchestra.run.smithi173.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 670 kB/s | 90 kB 00:00 2024-03-16T12:07:16.807 INFO:teuthology.orchestra.run.smithi173.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-16T12:07:16.824 INFO:teuthology.orchestra.run.smithi173.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-03-16T12:07:16.857 INFO:teuthology.orchestra.run.smithi173.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-16T12:07:17.041 INFO:teuthology.orchestra.run.smithi153.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-03-16T12:07:17.149 INFO:teuthology.orchestra.run.smithi153.stdout:(151/174): librados2-19.0.0-2162.g3d78f5ba.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-16T12:07:17.174 INFO:teuthology.orchestra.run.smithi173.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-03-16T12:07:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.4 MB/s | 3.1 MB 00:00 2024-03-16T12:07:17.241 INFO:teuthology.orchestra.run.smithi173.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-16T12:07:17.333 INFO:teuthology.orchestra.run.smithi153.stdout:(153/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.1 MB/s | 321 kB 00:00 2024-03-16T12:07:17.475 INFO:teuthology.orchestra.run.smithi153.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 299 kB/s | 42 kB 00:00 2024-03-16T12:07:17.550 INFO:teuthology.orchestra.run.smithi153.stdout:(155/174): NetworkManager-1.46.0-1.el9.x86_64.r 5.7 MB/s | 2.3 MB 00:00 2024-03-16T12:07:17.600 INFO:teuthology.orchestra.run.smithi153.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-16T12:07:17.634 INFO:teuthology.orchestra.run.smithi153.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.4 MB/s | 1.8 MB 00:00 2024-03-16T12:07:17.659 INFO:teuthology.orchestra.run.smithi153.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 620 kB/s | 36 kB 00:00 2024-03-16T12:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:(159/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-16T12:07:17.708 INFO:teuthology.orchestra.run.smithi173.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-03-16T12:07:17.712 INFO:teuthology.orchestra.run.smithi153.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.4 MB/s | 192 kB 00:00 2024-03-16T12:07:17.737 INFO:teuthology.orchestra.run.smithi153.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.2 MB/s | 256 kB 00:00 2024-03-16T12:07:17.762 INFO:teuthology.orchestra.run.smithi153.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-16T12:07:17.767 INFO:teuthology.orchestra.run.smithi173.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.4 MB/s | 2.3 MB 00:00 2024-03-16T12:07:17.787 INFO:teuthology.orchestra.run.smithi153.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-16T12:07:17.805 INFO:teuthology.orchestra.run.smithi098.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 363 kB/s | 465 kB 00:01 2024-03-16T12:07:17.808 INFO:teuthology.orchestra.run.smithi173.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-16T12:07:17.829 INFO:teuthology.orchestra.run.smithi153.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-16T12:07:17.846 INFO:teuthology.orchestra.run.smithi153.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-16T12:07:17.859 INFO:teuthology.orchestra.run.smithi173.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-03-16T12:07:17.863 INFO:teuthology.orchestra.run.smithi098.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 620 kB/s | 2.2 MB 00:03 2024-03-16T12:07:17.872 INFO:teuthology.orchestra.run.smithi153.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-16T12:07:17.888 INFO:teuthology.orchestra.run.smithi153.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 489 kB/s | 29 kB 00:00 2024-03-16T12:07:17.917 INFO:teuthology.orchestra.run.smithi173.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T12:07:17.968 INFO:teuthology.orchestra.run.smithi173.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 728 kB/s | 36 kB 00:00 2024-03-16T12:07:18.014 INFO:teuthology.orchestra.run.smithi153.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 1.2 MB 00:00 2024-03-16T12:07:18.018 INFO:teuthology.orchestra.run.smithi173.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-16T12:07:18.085 INFO:teuthology.orchestra.run.smithi173.stdout:(159/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-03-16T12:07:18.114 INFO:teuthology.orchestra.run.smithi153.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.6 MB/s | 465 kB 00:00 2024-03-16T12:07:18.144 INFO:teuthology.orchestra.run.smithi173.stdout:(160/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-16T12:07:18.156 INFO:teuthology.orchestra.run.smithi153.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 2.2 MB 00:00 2024-03-16T12:07:18.169 INFO:teuthology.orchestra.run.smithi173.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 276 kB/s | 23 kB 00:00 2024-03-16T12:07:18.181 INFO:teuthology.orchestra.run.smithi153.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-16T12:07:18.194 INFO:teuthology.orchestra.run.smithi173.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-03-16T12:07:18.219 INFO:teuthology.orchestra.run.smithi173.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-16T12:07:18.231 INFO:teuthology.orchestra.run.smithi153.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.7 MB/s | 2.3 MB 00:00 2024-03-16T12:07:18.245 INFO:teuthology.orchestra.run.smithi173.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-16T12:07:18.270 INFO:teuthology.orchestra.run.smithi173.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-16T12:07:18.295 INFO:teuthology.orchestra.run.smithi173.stdout:(166/174): libblkid-2.37.4-18.el9.x86_64.rpm 717 kB/s | 108 kB 00:00 2024-03-16T12:07:18.320 INFO:teuthology.orchestra.run.smithi173.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-16T12:07:18.357 INFO:teuthology.orchestra.run.smithi153.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-16T12:07:18.397 INFO:teuthology.orchestra.run.smithi098.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 624 kB/s | 2.3 MB 00:03 2024-03-16T12:07:18.415 INFO:teuthology.orchestra.run.smithi153.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 318 kB/s | 74 kB 00:00 2024-03-16T12:07:18.420 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-16T12:07:18.420 INFO:teuthology.orchestra.run.smithi153.stdout:Total 17 MB/s | 224 MB 00:12 2024-03-16T12:07:18.429 INFO:teuthology.orchestra.run.smithi173.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 1.2 MB 00:00 2024-03-16T12:07:18.504 INFO:teuthology.orchestra.run.smithi173.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.1 MB/s | 465 kB 00:00 2024-03-16T12:07:18.514 INFO:teuthology.orchestra.run.smithi098.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 44 kB/s | 31 kB 00:00 2024-03-16T12:07:18.555 INFO:teuthology.orchestra.run.smithi173.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-16T12:07:18.557 INFO:teuthology.orchestra.run.smithi098.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 469 kB/s | 74 kB 00:00 2024-03-16T12:07:18.605 INFO:teuthology.orchestra.run.smithi173.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.0 MB/s | 2.2 MB 00:00 2024-03-16T12:07:18.705 INFO:teuthology.orchestra.run.smithi173.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-16T12:07:18.855 INFO:teuthology.orchestra.run.smithi173.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.3 MB/s | 2.3 MB 00:00 2024-03-16T12:07:18.906 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-16T12:07:19.179 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-16T12:07:19.180 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-16T12:07:19.364 INFO:teuthology.orchestra.run.smithi173.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 98 kB/s | 74 kB 00:00 2024-03-16T12:07:19.371 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-16T12:07:19.371 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.3 MB/s | 224 MB 00:24 2024-03-16T12:07:19.832 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-16T12:07:20.105 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-16T12:07:20.105 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-16T12:07:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 487 kB/s | 1.1 MB 00:02 2024-03-16T12:07:20.131 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-16T12:07:20.131 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.4 MB/s | 224 MB 00:26 2024-03-16T12:07:20.900 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-16T12:07:20.901 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-16T12:07:21.556 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-16T12:07:21.822 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-16T12:07:21.822 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-16T12:07:21.931 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-16T12:07:21.931 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-16T12:07:24.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-16T12:07:24.274 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-16T12:07:27.797 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-16T12:07:27.935 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T12:07:27.978 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T12:07:28.008 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:28.018 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:28.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T12:07:28.112 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T12:07:28.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T12:07:28.171 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T12:07:28.213 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T12:07:28.590 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T12:07:28.622 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:28.744 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:28.765 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:28.856 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:28.877 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:07:28.895 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-16T12:07:28.917 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T12:07:29.080 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T12:07:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T12:07:29.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T12:07:29.135 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T12:07:29.161 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:29.174 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:29.223 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T12:07:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T12:07:29.304 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:29.310 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T12:07:29.351 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:29.353 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T12:07:29.391 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T12:07:29.395 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T12:07:29.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:29.430 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:29.621 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:29.653 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:29.684 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T12:07:29.717 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T12:07:29.771 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:29.788 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T12:07:29.887 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:29.891 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:29.913 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:29.931 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:29.992 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:30.003 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:07:30.039 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T12:07:30.240 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T12:07:30.247 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:30.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T12:07:30.327 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:30.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T12:07:30.421 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:30.439 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T12:07:30.444 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:30.478 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T12:07:30.491 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:30.504 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T12:07:30.540 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T12:07:30.548 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T12:07:30.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:30.586 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:30.614 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T12:07:30.668 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T12:07:30.706 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T12:07:30.754 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:30.771 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T12:07:30.785 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:30.805 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T12:07:30.817 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T12:07:30.849 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T12:07:30.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T12:07:30.910 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T12:07:30.923 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T12:07:30.982 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T12:07:30.994 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:31.017 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T12:07:31.056 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:31.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T12:07:31.089 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T12:07:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:31.156 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T12:07:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:31.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T12:07:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T12:07:31.367 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:31.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T12:07:31.461 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T12:07:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T12:07:31.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T12:07:31.602 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T12:07:31.629 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T12:07:31.656 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T12:07:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-16T12:07:31.730 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T12:07:31.783 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T12:07:31.822 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T12:07:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T12:07:31.887 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T12:07:31.888 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T12:07:31.921 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T12:07:31.925 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T12:07:31.969 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T12:07:31.978 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T12:07:31.997 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T12:07:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T12:07:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T12:07:32.056 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T12:07:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T12:07:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T12:07:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T12:07:32.174 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T12:07:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:32.182 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T12:07:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:32.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T12:07:32.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T12:07:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T12:07:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T12:07:32.265 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T12:07:32.270 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T12:07:32.303 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T12:07:32.317 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T12:07:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T12:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T12:07:32.390 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T12:07:32.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T12:07:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T12:07:32.578 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T12:07:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T12:07:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T12:07:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T12:07:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T12:07:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:07:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:33.096 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T12:07:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:07:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:07:33.173 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T12:07:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T12:07:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T12:07:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T12:07:33.267 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T12:07:33.310 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:33.317 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T12:07:33.350 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T12:07:33.393 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T12:07:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T12:07:33.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T12:07:33.443 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T12:07:33.460 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T12:07:33.475 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T12:07:33.507 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T12:07:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T12:07:33.543 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T12:07:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T12:07:33.586 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T12:07:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:33.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T12:07:33.715 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T12:07:33.759 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:07:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T12:07:33.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T12:07:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T12:07:33.875 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T12:07:33.878 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T12:07:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T12:07:33.921 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T12:07:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T12:07:33.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:33.991 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T12:07:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T12:07:34.013 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T12:07:34.038 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T12:07:34.078 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T12:07:34.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T12:07:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T12:07:34.128 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T12:07:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T12:07:34.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T12:07:34.229 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T12:07:34.309 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T12:07:34.314 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:34.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T12:07:34.341 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T12:07:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T12:07:34.378 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T12:07:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T12:07:34.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T12:07:34.425 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T12:07:34.461 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T12:07:34.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T12:07:34.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T12:07:34.566 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T12:07:34.640 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T12:07:34.696 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T12:07:34.750 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T12:07:34.799 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T12:07:34.813 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T12:07:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:34.936 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T12:07:34.980 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T12:07:35.012 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T12:07:35.015 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T12:07:35.039 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T12:07:35.060 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T12:07:35.088 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T12:07:35.093 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T12:07:35.122 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T12:07:35.123 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T12:07:35.161 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T12:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T12:07:35.193 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T12:07:35.202 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:35.223 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T12:07:35.259 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T12:07:35.270 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:35.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T12:07:35.299 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T12:07:35.333 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T12:07:35.339 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T12:07:35.364 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T12:07:35.366 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T12:07:35.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T12:07:35.407 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T12:07:35.442 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T12:07:35.461 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T12:07:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T12:07:35.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T12:07:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T12:07:35.623 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T12:07:35.664 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T12:07:35.680 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T12:07:35.696 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T12:07:35.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T12:07:35.743 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T12:07:35.771 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T12:07:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T12:07:35.826 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T12:07:35.841 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T12:07:35.889 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T12:07:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T12:07:35.970 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T12:07:36.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T12:07:36.010 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T12:07:36.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T12:07:36.068 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T12:07:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T12:07:36.205 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T12:07:36.241 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T12:07:36.247 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T12:07:36.273 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T12:07:36.304 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T12:07:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T12:07:36.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T12:07:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T12:07:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T12:07:36.417 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T12:07:36.438 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T12:07:36.468 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T12:07:36.468 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T12:07:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T12:07:36.535 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T12:07:36.569 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T12:07:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T12:07:36.603 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T12:07:36.617 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T12:07:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T12:07:36.644 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T12:07:36.669 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T12:07:36.679 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T12:07:36.695 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T12:07:36.734 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T12:07:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T12:07:36.768 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T12:07:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T12:07:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T12:07:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T12:07:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T12:07:36.849 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T12:07:36.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T12:07:36.893 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T12:07:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T12:07:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T12:07:36.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T12:07:37.024 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T12:07:37.060 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T12:07:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T12:07:37.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T12:07:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T12:07:37.138 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T12:07:37.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T12:07:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T12:07:37.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T12:07:37.781 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T12:07:37.858 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T12:07:37.891 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T12:07:37.914 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T12:07:37.948 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T12:07:37.996 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T12:07:38.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T12:07:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T12:07:38.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T12:07:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T12:07:38.293 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T12:07:38.322 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T12:07:38.333 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T12:07:38.377 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:38.402 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T12:07:38.428 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T12:07:38.448 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T12:07:38.509 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T12:07:38.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T12:07:38.618 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T12:07:39.027 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T12:07:39.066 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T12:07:39.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T12:07:39.177 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T12:07:39.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T12:07:39.233 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T12:07:39.264 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T12:07:39.268 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T12:07:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T12:07:39.408 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T12:07:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T12:07:39.594 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T12:07:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T12:07:39.712 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T12:07:39.720 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:39.742 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T12:07:39.779 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T12:07:39.787 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T12:07:39.891 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T12:07:39.938 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T12:07:39.978 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T12:07:40.046 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T12:07:40.085 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T12:07:40.146 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T12:07:40.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T12:07:40.238 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T12:07:40.282 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:40.354 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:40.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T12:07:40.392 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T12:07:40.395 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T12:07:40.489 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T12:07:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T12:07:40.520 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T12:07:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T12:07:40.551 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T12:07:40.578 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T12:07:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T12:07:40.738 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T12:07:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T12:07:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T12:07:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T12:07:40.892 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T12:07:40.894 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T12:07:40.917 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:40.990 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:41.021 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T12:07:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T12:07:41.058 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T12:07:41.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T12:07:41.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T12:07:41.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T12:07:41.475 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T12:07:41.520 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T12:07:41.546 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:41.549 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T12:07:41.599 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:41.622 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T12:07:41.827 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T12:07:41.884 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T12:07:42.057 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T12:07:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T12:07:42.151 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T12:07:42.198 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T12:07:42.238 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T12:07:42.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T12:07:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T12:07:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T12:07:42.461 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T12:07:42.565 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:42.597 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:42.618 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T12:07:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T12:07:42.725 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T12:07:42.807 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T12:07:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T12:07:43.001 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T12:07:43.117 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T12:07:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:43.167 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:43.173 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T12:07:43.210 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T12:07:43.248 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:43.248 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T12:07:43.248 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T12:07:43.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T12:07:43.276 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:07:43.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T12:07:43.483 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T12:07:43.864 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:43.901 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:44.434 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T12:07:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:44.517 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T12:07:44.551 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T12:07:44.553 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:44.553 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T12:07:44.553 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T12:07:44.553 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:44.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:07:44.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T12:07:44.732 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T12:07:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T12:07:45.019 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T12:07:45.081 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T12:07:45.185 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T12:07:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T12:07:46.141 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T12:07:47.650 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T12:07:47.745 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T12:07:47.783 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T12:07:47.813 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T12:07:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T12:07:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T12:07:48.261 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T12:07:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T12:07:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T12:07:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T12:07:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T12:07:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T12:07:51.791 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:52.455 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T12:07:52.500 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:52.648 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T12:07:52.648 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T12:07:52.648 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T12:07:52.648 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:07:52.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:07:54.986 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:07:54.986 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-16T12:07:54.986 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-16T12:07:54.986 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-16T12:07:54.987 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-16T12:07:54.987 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-16T12:07:54.987 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-16T12:07:54.987 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-16T12:07:54.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:55.153 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:07:55.169 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:07:55.169 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:55.169 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T12:07:55.169 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:56.508 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:07:56.508 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-16T12:07:56.509 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:56.666 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:07:56.675 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T12:07:56.691 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:07:56.692 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:07:56.692 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:56.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:07:56.947 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:07:56.947 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:56.947 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T12:07:56.947 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:07:56.948 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:07:56.948 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:57.392 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:07:57.415 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:57.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T12:07:57.930 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:07:57.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:07:57.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T12:07:58.053 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T12:07:58.083 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:07:58.188 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:07:58.214 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:58.319 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:07:58.359 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:07:58.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:07:58.454 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:07:58.455 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:58.455 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T12:07:58.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:07:58.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:07:58.455 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:07:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:07:58.912 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:58.912 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T12:07:58.912 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:07:58.913 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:07:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:07:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T12:07:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:07:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:07:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:07:59.454 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:07:59.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:07:59.497 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T12:07:59.548 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T12:07:59.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:07:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T12:07:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:07:59.676 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:07:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:07:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T12:07:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:07:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:07:59.677 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:07:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:07:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:07:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T12:07:59.806 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:07:59.839 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:08:00.648 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T12:08:00.681 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:00.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:00.748 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:00.770 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:00.770 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:00.770 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T12:08:00.770 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:00.805 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:08:00.815 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:00.838 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:00.839 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:08:00.939 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:08:01.094 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T12:08:01.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:08:01.139 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:01.149 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:01.172 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:01.172 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:01.172 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T12:08:01.172 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:01.172 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:01.173 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:01.184 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:08:01.194 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:08:01.239 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T12:08:02.114 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T12:08:02.154 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:02.177 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:02.177 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:02.177 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T12:08:02.177 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:02.178 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:02.178 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:02.222 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:02.244 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:02.244 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:02.244 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T12:08:02.244 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:02.289 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:02.311 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:02.312 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:02.312 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T12:08:02.312 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:02.312 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:02.312 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:05.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:05.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T12:08:05.701 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T12:08:05.712 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-16T12:08:05.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:05.728 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T12:08:05.912 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T12:08:05.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:08:06.013 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T12:08:06.013 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:06.013 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T12:08:06.013 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:06.035 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T12:08:06.083 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T12:08:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T12:08:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T12:08:06.218 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T12:08:06.252 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T12:08:06.292 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T12:08:06.321 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T12:08:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T12:08:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T12:08:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:07.110 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T12:08:07.117 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:07.140 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T12:08:07.150 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:07.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T12:08:07.171 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:07.348 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T12:08:07.475 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T12:08:07.515 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T12:08:07.554 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T12:08:07.607 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T12:08:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:08:07.650 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T12:08:07.683 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T12:08:07.701 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:07.715 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T12:08:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T12:08:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T12:08:07.877 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T12:08:07.910 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T12:08:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:08:08.011 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T12:08:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T12:08:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T12:08:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:08:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T12:08:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:08.095 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T12:08:08.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T12:08:08.217 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T12:08:08.257 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T12:08:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T12:08:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T12:08:09.970 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T12:08:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T12:08:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:11.149 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T12:08:11.190 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:08:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T12:08:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T12:08:11.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:08:11.619 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T12:08:11.620 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:11.620 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T12:08:11.620 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:08:11.620 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T12:08:11.620 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:08:11.714 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T12:08:11.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T12:08:12.800 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T12:08:12.837 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T12:08:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:13.031 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T12:08:13.031 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:13.031 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T12:08:13.032 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:13.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T12:08:13.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:13.572 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T12:08:13.641 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T12:08:13.641 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T12:08:13.641 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:13.642 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T12:08:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:08:14.150 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T12:08:14.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T12:08:14.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T12:08:14.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T12:08:14.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T12:08:14.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T12:08:14.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T12:08:14.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T12:08:14.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T12:08:14.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T12:08:14.167 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T12:08:14.168 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T12:08:14.169 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T12:08:14.170 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T12:08:14.171 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T12:08:14.172 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T12:08:14.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T12:08:14.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T12:08:14.175 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T12:08:14.176 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T12:08:14.177 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T12:08:14.178 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:14.179 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T12:08:14.179 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T12:08:14.179 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T12:08:15.628 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:15.628 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T12:08:15.629 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T12:08:15.630 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T12:08:15.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T12:08:15.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T12:08:15.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T12:08:15.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T12:08:15.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T12:08:15.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T12:08:15.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T12:08:15.638 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T12:08:15.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T12:08:15.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T12:08:15.641 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T12:08:15.642 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T12:08:17.163 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T12:08:17.164 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.165 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T12:08:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T12:08:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T12:08:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T12:08:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T12:08:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T12:08:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T12:08:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T12:08:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-16T12:08:17.176 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-16T12:08:17.614 DEBUG:teuthology.parallel:result is None 2024-03-16T12:08:18.411 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:18.455 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T12:08:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T12:08:18.562 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T12:08:18.562 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.563 INFO:teuthology.orchestra.run.smithi173.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T12:08:18.564 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.565 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T12:08:18.566 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.567 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T12:08:18.568 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T12:08:18.569 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T12:08:18.570 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T12:08:18.571 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T12:08:18.572 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T12:08:18.573 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T12:08:18.574 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:08:18.575 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-16T12:08:18.832 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T12:08:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T12:08:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T12:08:19.044 DEBUG:teuthology.parallel:result is None 2024-03-16T12:08:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T12:08:19.170 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T12:08:19.217 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T12:08:19.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T12:08:19.312 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T12:08:19.364 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T12:08:19.450 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T12:08:19.554 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T12:08:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T12:08:20.529 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T12:08:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T12:08:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T12:08:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T12:08:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T12:08:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T12:08:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T12:08:22.311 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T12:08:22.369 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T12:08:22.370 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T12:08:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T12:08:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:22.711 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T12:08:22.755 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T12:08:22.801 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T12:08:22.835 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T12:08:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T12:08:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T12:08:23.013 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T12:08:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T12:08:23.089 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T12:08:23.153 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T12:08:23.198 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T12:08:23.275 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:23.275 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:23.597 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T12:08:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:23.762 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T12:08:23.793 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T12:08:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T12:08:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T12:08:24.047 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:24.308 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T12:08:24.309 INFO:teuthology.orchestra.run.smithi098.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-16T12:08:24.309 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T12:08:29.349 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T12:08:29.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T12:08:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T12:08:29.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T12:08:29.367 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T12:08:29.368 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T12:08:29.369 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T12:08:29.370 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T12:08:29.371 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T12:08:29.372 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T12:08:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T12:08:29.374 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T12:08:29.375 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T12:08:29.376 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T12:08:29.377 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T12:08:29.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T12:08:31.982 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T12:08:31.982 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T12:08:31.983 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:31.984 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.985 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.986 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.987 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T12:08:31.988 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.989 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T12:08:31.990 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T12:08:31.991 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T12:08:31.992 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T12:08:31.993 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T12:08:31.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T12:08:31.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T12:08:31.996 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T12:08:31.997 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T12:08:31.998 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T12:08:31.999 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T12:08:31.999 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:08:31.999 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-16T12:08:32.631 DEBUG:teuthology.parallel:result is None 2024-03-16T12:08:32.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:32.855 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T12:08:32.913 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:32.913 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:32.913 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T12:08:32.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:33.134 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T12:08:33.161 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:33.161 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:33.161 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T12:08:33.163 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:33.306 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T12:08:33.332 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:33.332 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T12:08:33.332 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T12:08:33.337 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T12:08:33.338 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:08:33.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T12:08:33.424 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:08:33.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T12:08:33.459 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:08:33.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T12:08:33.493 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T12:08:33.493 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:08:33.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T12:08:33.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T12:08:33.683 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:08:33.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T12:08:33.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T12:08:33.784 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:08:33.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T12:08:33.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T12:08:33.884 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T12:08:33.884 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:08:33.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T12:08:33.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T12:08:34.074 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:08:34.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T12:08:34.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T12:08:34.171 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:08:34.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T12:08:34.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T12:08:34.265 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T12:08:34.266 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:08:34.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T12:08:34.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T12:08:34.458 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:08:34.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T12:08:34.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T12:08:34.566 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:08:34.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T12:08:34.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T12:08:34.663 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T12:08:34.766 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-16T12:08:34.766 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:34.767 INFO:tasks.cephadm:Cluster fsid is e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:08:34.767 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T12:08:34.767 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.153', 'mon.c': '172.21.15.173'} 2024-03-16T12:08:34.767 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-03-16T12:08:34.767 INFO:tasks.cephadm:First mgr is x 2024-03-16T12:08:34.768 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T12:08:34.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-16T12:08:34.841 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-03-16T12:08:34.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-03-16T12:08:34.896 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T12:08:34.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:35.054 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T12:08:35.275 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:08:35.277 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-16T12:08:35.277 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:35.902 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 12:08 /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:35.903 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:36.648 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 12:08 /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:36.648 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:37.238 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 12:08 /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:37.239 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:37.301 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:37.320 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T12:08:37.354 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-16T12:08:37.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T12:08:37.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T12:08:37.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T12:08:38.153 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T12:08:38.193 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T12:08:38.544 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T12:09:10.847 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-16T12:09:10.848 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T12:09:10.848 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T12:09:10.848 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-03-16T12:09:10.848 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T12:09:10.848 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-16T12:09:10.849 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-03-16T12:09:11.216 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T12:09:12.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:09:12.693 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:09:12.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-16T12:09:12.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-03-16T12:09:12.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-03-16T12:09:12.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-16T12:09:13.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-03-16T12:09:13.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-03-16T12:09:13.064 INFO:tasks.cephadm:Writing seed config... 2024-03-16T12:09:13.065 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T12:09:13.065 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T12:09:13.065 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T12:09:13.065 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T12:09:13.065 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-16T12:09:13.066 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T12:09:13.067 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T12:09:13.068 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-16T12:09:13.069 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:09:13.069 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T12:09:13.125 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e9cfddf2-e38d-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T12:09:13.125 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T12:09:13.172 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T12:09:13.218 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T12:09:13.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:09:13.494 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-16T12:09:13.495 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'e9cfddf2-e38d-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-03-16T12:09:13.551 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T12:09:13.552 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T12:09:13.552 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-03-16T12:09:13.603 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T12:09:13.604 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-03-16T12:09:13.604 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-03-16T12:09:13.620 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T12:09:13.620 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T12:09:13.638 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T12:09:13.638 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:09:13.661 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-03-16T12:09:13.679 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-03-16T12:09:13.680 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-03-16T12:09:13.680 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-03-16T12:09:13.733 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T12:09:13.733 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T12:09:13.733 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-03-16T12:09:13.733 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-03-16T12:09:13.747 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T12:09:13.747 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T12:09:13.760 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T12:09:13.760 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:09:13.776 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-03-16T12:09:13.796 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-03-16T12:09:13.796 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-03-16T12:09:13.796 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-03-16T12:09:13.796 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:13.797 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 140391983013312 on /run/cephadm/e9cfddf2-e38d-11ee-95c9-87774f69a715.lock 2024-03-16T12:09:13.797 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 140391983013312 acquired on /run/cephadm/e9cfddf2-e38d-11ee-95c9-87774f69a715.lock 2024-03-16T12:09:13.798 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-03-16T12:09:13.799 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-03-16T12:09:13.800 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-16T12:09:13.813 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-03-16T12:09:13.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-03-16T12:09:13.826 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T12:09:13.826 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-16T12:09:13.826 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2024-03-16T12:09:13.836 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T12:09:13.837 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T12:09:13.837 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T12:09:13.837 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-16T12:09:13.837 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-03-16T12:09:13.837 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T12:09:13.839 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-03-16T12:09:13.839 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-03-16T12:09:13.840 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T12:09:13.842 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T12:09:13.844 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T12:09:14.491 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T12:09:14.492 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T12:09:14.492 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T12:09:14.492 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T12:09:14.493 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-16T12:09:14.493 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T12:09:14.493 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T12:09:16.042 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T12:09:16.043 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T12:09:16.043 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T12:09:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-03-16T12:09:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-03-16T12:09:18.328 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDti/Vlnn6OJxAA84xrK1eKK4oVQkAj4TKzkQ== 2024-03-16T12:09:19.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDui/Vl3JViMBAAlEw2EJmTXcUQqusw25iRnA== 2024-03-16T12:09:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQDvi/Vl7REnNxAATDtv3IocHE2qnwNuMNn+MQ== 2024-03-16T12:09:20.512 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-03-16T12:09:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:21.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T12:09:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:09:21.670 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-03-16T12:09:32.207 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-03-16T12:09:32.585 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T12:09:32.818 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T12:09:33.119 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target. 2024-03-16T12:09:33.119 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target. 2024-03-16T12:09:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a 2024-03-16T12:09:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T12:09:33.781 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target.wants/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@.service. 2024-03-16T12:09:34.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 podman[31247]: 2024-03-16 12:09:34.105245045 +0000 UTC m=+0.035098281 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:09:34.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 podman[31247]: 2024-03-16 12:09:34.665894131 +0000 UTC m=+0.595747371 container create 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T12:09:34.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 podman[31247]: 2024-03-16 12:09:34.895171498 +0000 UTC m=+0.825024775 container init 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T12:09:34.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 podman[31247]: 2024-03-16 12:09:34.910188519 +0000 UTC m=+0.840041758 container start 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T12:09:35.002 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T12:09:35.002 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-16T12:09:35.017 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T12:09:35.017 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:09:35.017 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-16T12:09:35.017 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T12:09:35.018 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-03-16T12:09:35.018 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-03-16T12:09:35.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 bash[31247]: 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 2024-03-16T12:09:35.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 ceph-mon[31265]: mkfs e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:35.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 ceph-mon[31265]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T12:09:35.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:34 smithi098 systemd[1]: Started Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:09:36.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: monmap epoch 1 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: last_changed 2024-03-16T12:09:20.978440+0000 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: min_mon_release 19 (squid) 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: election_strategy: 1 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: fsmap 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: mgrmap e1: no daemons active 2024-03-16T12:09:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:35 smithi098 ceph-mon[31265]: from='client.? 172.21.15.98:0/1401164802' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-16T12:09:36.408 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T12:09:36.409 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:36.409 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T12:09:36.409 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:36.409 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-03-16T12:09:36.409 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.76711s) 2024-03-16T12:09:36.410 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T12:09:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.658+0000 7fe756600700 1 Processor -- start 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.658+0000 7fe756600700 1 -- start start 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.658+0000 7fe756600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.658+0000 7fe756600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe75010a7f0 con 0x7fe750109e00 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.659+0000 7fe74f400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.659+0000 7fe74f400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49532/0 (socket says 172.21.15.98:49532) 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.659+0000 7fe74f400700 1 -- 172.21.15.98:0/1864908982 learned_addr learned my addr 172.21.15.98:0/1864908982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.660+0000 7fe74f400700 1 -- 172.21.15.98:0/1864908982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75010afb0 con 0x7fe750109e00 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.660+0000 7fe74f400700 1 --2- 172.21.15.98:0/1864908982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3792110b4ef28c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe74ea00700 1 -- 172.21.15.98:0/1864908982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740004160 con 0x7fe750109e00 2024-03-16T12:09:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe74ea00700 1 -- 172.21.15.98:0/1864908982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe740004300 con 0x7fe750109e00 2024-03-16T12:09:36.414 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe74ea00700 1 -- 172.21.15.98:0/1864908982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7400036a0 con 0x7fe750109e00 2024-03-16T12:09:36.414 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe756600700 1 -- 172.21.15.98:0/1864908982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 msgr2=0x7fe75010a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:36.414 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe756600700 1 --2- 172.21.15.98:0/1864908982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe740009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.661+0000 7fe756600700 1 -- 172.21.15.98:0/1864908982 shutdown_connections 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.662+0000 7fe756600700 1 --2- 172.21.15.98:0/1864908982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75010a220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.662+0000 7fe756600700 1 -- 172.21.15.98:0/1864908982 >> 172.21.15.98:0/1864908982 conn(0x7fe75007d1d0 msgr2=0x7fe75007d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.662+0000 7fe756600700 1 -- 172.21.15.98:0/1864908982 shutdown_connections 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.662+0000 7fe756600700 1 -- 172.21.15.98:0/1864908982 wait complete. 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe756600700 1 Processor -- start 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe756600700 1 -- start start 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe756600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe756600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe75010b4e0 con 0x7fe750109e00 2024-03-16T12:09:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe74f400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe74f400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49546/0 (socket says 172.21.15.98:49546) 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.663+0000 7fe74f400700 1 -- 172.21.15.98:0/1401164802 learned_addr learned my addr 172.21.15.98:0/1401164802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe74f400700 1 -- 172.21.15.98:0/1401164802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75019d9e0 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe74f400700 1 --2- 172.21.15.98:0/1401164802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe740000c00 tx=0x7fe740004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740003db0 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe740011630 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75019dc70 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.664+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740011950 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.665+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7501a0960 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.665+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe74001a460 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.665+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe74001b780 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.666+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe750063900 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.669+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe74001b980 con 0x7fe750109e00 2024-03-16T12:09:36.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.748+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fe75002ddc0 con 0x7fe750109e00 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.748+0000 7fe74cc00700 1 -- 172.21.15.98:0/1401164802 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7fe74001bb60 con 0x7fe750109e00 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.749+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 msgr2=0x7fe75019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.749+0000 7fe756600700 1 --2- 172.21.15.98:0/1401164802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe740000c00 tx=0x7fe740004750 comp rx=0 tx=0).stop 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.750+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 shutdown_connections 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.750+0000 7fe756600700 1 --2- 172.21.15.98:0/1401164802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe750109e00 0x7fe75019d4a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.750+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 >> 172.21.15.98:0/1401164802 conn(0x7fe75007d1d0 msgr2=0x7fe750194110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.750+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 shutdown_connections 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:35.750+0000 7fe756600700 1 -- 172.21.15.98:0/1401164802 wait complete. 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-03-16T12:09:36.420 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T12:09:37.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:37 smithi098 ceph-mon[31265]: from='client.? 172.21.15.98:0/2265054513' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T12:09:37.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:37 smithi098 ceph-mon[31265]: from='client.? 172.21.15.98:0/2265054513' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T12:09:38.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:38.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T12:09:38.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:38.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-16T12:09:38.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T12:09:38.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T12:09:38.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.474+0000 7fe0c3600700 1 Processor -- start 2024-03-16T12:09:38.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.474+0000 7fe0c3600700 1 -- start start 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.474+0000 7fe0c3600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.474+0000 7fe0c3600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc07be90 con 0x7fe0bc108630 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.475+0000 7fe0c0c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.475+0000 7fe0c0c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49748/0 (socket says 172.21.15.98:49748) 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.475+0000 7fe0c0c00700 1 -- 172.21.15.98:0/1927666379 learned_addr learned my addr 172.21.15.98:0/1927666379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.475+0000 7fe0c0c00700 1 -- 172.21.15.98:0/1927666379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc10af60 con 0x7fe0bc108630 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.476+0000 7fe0c0c00700 1 --2- 172.21.15.98:0/1927666379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00098a0 tx=0x7fe0b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c27f53f1617450 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:38.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.476+0000 7fe0bb400700 1 -- 172.21.15.98:0/1927666379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0004160 con 0x7fe0bc108630 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.476+0000 7fe0bb400700 1 -- 172.21.15.98:0/1927666379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b0004300 con 0x7fe0bc108630 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.476+0000 7fe0bb400700 1 -- 172.21.15.98:0/1927666379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b00036a0 con 0x7fe0bc108630 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 -- 172.21.15.98:0/1927666379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 msgr2=0x7fe0bc10aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 --2- 172.21.15.98:0/1927666379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe0b00098a0 tx=0x7fe0b0009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 -- 172.21.15.98:0/1927666379 shutdown_connections 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 --2- 172.21.15.98:0/1927666379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc10aa20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 -- 172.21.15.98:0/1927666379 >> 172.21.15.98:0/1927666379 conn(0x7fe0bc1020a0 msgr2=0x7fe0bc1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:38.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 -- 172.21.15.98:0/1927666379 shutdown_connections 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.477+0000 7fe0c3600700 1 -- 172.21.15.98:0/1927666379 wait complete. 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.478+0000 7fe0c3600700 1 Processor -- start 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.478+0000 7fe0c3600700 1 -- start start 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.479+0000 7fe0c3600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.479+0000 7fe0c3600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc10b430 con 0x7fe0bc108630 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.479+0000 7fe0c0c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0c0c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49764/0 (socket says 172.21.15.98:49764) 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0c0c00700 1 -- 172.21.15.98:0/2265054513 learned_addr learned my addr 172.21.15.98:0/2265054513 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0c0c00700 1 -- 172.21.15.98:0/2265054513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc19d9b0 con 0x7fe0bc108630 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0c0c00700 1 --2- 172.21.15.98:0/2265054513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0000c00 tx=0x7fe0b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0003db0 con 0x7fe0bc108630 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b0011630 con 0x7fe0bc108630 2024-03-16T12:09:38.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0b0011950 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.480+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0bc19dc40 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.481+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0bc1a0930 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.481+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe0b001a460 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.481+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe0b001b780 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.482+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0bc063900 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.485+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe0b001b980 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.561+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe0bc02ddc0 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.568+0000 7fe0b9600700 1 -- 172.21.15.98:0/2265054513 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fe0b001bb60 con 0x7fe0bc108630 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.569+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 msgr2=0x7fe0bc19d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.569+0000 7fe0c3600700 1 --2- 172.21.15.98:0/2265054513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0000c00 tx=0x7fe0b0004750 comp rx=0 tx=0).stop 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.570+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 shutdown_connections 2024-03-16T12:09:38.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.570+0000 7fe0c3600700 1 --2- 172.21.15.98:0/2265054513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0bc108630 0x7fe0bc19d470 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:38.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.570+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 >> 172.21.15.98:0/2265054513 conn(0x7fe0bc1020a0 msgr2=0x7fe0bc102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:38.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.570+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 shutdown_connections 2024-03-16T12:09:38.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:37.570+0000 7fe0c3600700 1 -- 172.21.15.98:0/2265054513 wait complete. 2024-03-16T12:09:38.266 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-03-16T12:09:39.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:39 smithi098 ceph-mon[31265]: from='client.? 172.21.15.98:0/1733683755' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:09:39.663 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.899+0000 7f7f23800700 1 Processor -- start 2024-03-16T12:09:39.663 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f23800700 1 -- start start 2024-03-16T12:09:39.663 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f23800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f23800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c082430 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f20e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f20e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49778/0 (socket says 172.21.15.98:49778) 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.900+0000 7f7f20e00700 1 -- 172.21.15.98:0/3613081552 learned_addr learned my addr 172.21.15.98:0/3613081552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.901+0000 7f7f20e00700 1 -- 172.21.15.98:0/3613081552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c1115e0 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.901+0000 7f7f20e00700 1 --2- 172.21.15.98:0/3613081552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098a0 tx=0x7f7f10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d158daad7b9965e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:39.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.901+0000 7f7f1b400700 1 -- 172.21.15.98:0/3613081552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f10004160 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.901+0000 7f7f1b400700 1 -- 172.21.15.98:0/3613081552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f10004300 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.901+0000 7f7f1b400700 1 -- 172.21.15.98:0/3613081552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f10011b50 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 -- 172.21.15.98:0/3613081552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 msgr2=0x7f7f1c1110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 --2- 172.21.15.98:0/3613081552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098a0 tx=0x7f7f10009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 -- 172.21.15.98:0/3613081552 shutdown_connections 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 --2- 172.21.15.98:0/3613081552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1110a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 -- 172.21.15.98:0/3613081552 >> 172.21.15.98:0/3613081552 conn(0x7f7f1c07d1d0 msgr2=0x7f7f1c07f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 -- 172.21.15.98:0/3613081552 shutdown_connections 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.902+0000 7f7f23800700 1 -- 172.21.15.98:0/3613081552 wait complete. 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.903+0000 7f7f23800700 1 Processor -- start 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f23800700 1 -- start start 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f23800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f23800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c112530 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f20e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f20e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49794/0 (socket says 172.21.15.98:49794) 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.904+0000 7f7f20e00700 1 -- 172.21.15.98:0/1733683755 learned_addr learned my addr 172.21.15.98:0/1733683755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f20e00700 1 -- 172.21.15.98:0/1733683755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c1a26c0 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f20e00700 1 --2- 172.21.15.98:0/1733683755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f10011b60 tx=0x7f7f10011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f10003970 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f10003b10 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f10024610 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1c1a2950 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.905+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1c1a5640 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.906+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f10024c50 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.906+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f10011e00 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.907+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f1c063900 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.910+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7f1001a770 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.985+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7f1c1a5810 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.986+0000 7f7f19600700 1 -- 172.21.15.98:0/1733683755 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f7f1001a9f0 con 0x7f7f1c10ecb0 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 msgr2=0x7f7f1c1a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 --2- 172.21.15.98:0/1733683755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f10011b60 tx=0x7f7f10011c40 comp rx=0 tx=0).stop 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 shutdown_connections 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 --2- 172.21.15.98:0/1733683755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f1c10ecb0 0x7f7f1c1a2180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 >> 172.21.15.98:0/1733683755 conn(0x7f7f1c07d1d0 msgr2=0x7f7f1c07dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.987+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 shutdown_connections 2024-03-16T12:09:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:38.988+0000 7f7f23800700 1 -- 172.21.15.98:0/1733683755 wait complete. 2024-03-16T12:09:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-03-16T12:09:40.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:39 smithi098 systemd[1]: Stopping Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:09:40.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a[31261]: 2024-03-16T12:09:40.108+0000 7fce87800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:40.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a[31261]: 2024-03-16T12:09:40.108+0000 7fce87800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T12:09:40.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 podman[31509]: 2024-03-16 12:09:40.171493194 +0000 UTC m=+0.191403944 container died 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:40.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 podman[31509]: 2024-03-16 12:09:40.552415668 +0000 UTC m=+0.572326414 container cleanup 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T12:09:40.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 bash[31509]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a 2024-03-16T12:09:41.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:40 smithi098 podman[31523]: 2024-03-16 12:09:40.6951381 +0000 UTC m=+0.506792152 container remove 56cca9526262415ce251fe6ff3a7c1285c7f774985665c071d26d9eb6762b643 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T12:09:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 systemd[1]: Stopped Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:09:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service: Consumed 1.860s CPU time. 2024-03-16T12:09:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 systemd[1]: Starting Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:09:42.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 podman[31614]: 2024-03-16 12:09:41.479317288 +0000 UTC m=+0.042748899 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:09:42.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:41 smithi098 podman[31614]: 2024-03-16 12:09:41.629624627 +0000 UTC m=+0.193056236 container create 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T12:09:42.292 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 podman[31614]: 2024-03-16 12:09:42.225275871 +0000 UTC m=+0.788707484 container init 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 podman[31614]: 2024-03-16 12:09:42.235425843 +0000 UTC m=+0.798857422 container start 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: pidfile_write: ignore empty --pid-file 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: load: jerasure load: lrc 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: RocksDB version: 7.9.2 2024-03-16T12:09:42.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Git sha 0 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: DB SUMMARY 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: DB Session ID: 92G0TSENW7IGPAZR1R5K 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: CURRENT file: CURRENT 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103768 ; 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.error_if_exists: 0 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.create_if_missing: 0 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.paranoid_checks: 1 2024-03-16T12:09:42.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.env: 0x559d9e030c60 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.info_log: 0x559d9eb2db00 2024-03-16T12:09:42.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.statistics: (nil) 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.use_fsync: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_log_file_size: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_fallocate: 1 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T12:09:42.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.use_direct_reads: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.db_log_dir: 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.wal_dir: 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.write_buffer_manager: 0x559d9eb001e0 2024-03-16T12:09:42.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.unordered_write: 0 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T12:09:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.row_cache: None 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.wal_filter: None 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.two_write_queues: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.wal_compression: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.atomic_flush: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T12:09:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.log_readahead_size: 0 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_background_jobs: 2 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_background_compactions: -1 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_subcompactions: 1 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T12:09:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 bash[31614]: 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_open_files: -1 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_background_flushes: -1 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Compression algorithms supported: 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T12:09:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kZSTD supported: 0 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kXpressCompression supported: 0 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kZlibCompression supported: 1 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kSnappyCompression supported: 1 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kLZ4Compression supported: 1 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: kBZip2Compression supported: 0 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.merge_operator: 2024-03-16T12:09:42.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_filter: None 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559d9eb2dc20) 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-03-16T12:09:42.459 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x559d9eb33090 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-03-16T12:09:42.460 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-03-16T12:09:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-03-16T12:09:42.462 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression: NoCompression 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.num_levels: 7 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T12:09:42.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T12:09:42.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T12:09:42.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T12:09:42.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.table_properties_collectors: 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.inplace_update_support: 0 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T12:09:42.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.bloom_locality: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.max_successive_merges: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.ttl: 2592000 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T12:09:42.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enable_blob_files: false 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.min_blob_size: 0 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 62ca8662-215c-4538-a964-29a03c2405ce 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710590982289263, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T12:09:42.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710590982290871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98971, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96950, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13836, "raw_average_key_size": 49, "raw_value_size": 89407, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710590982, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "62ca8662-215c-4538-a964-29a03c2405ce", "db_session_id": "92G0TSENW7IGPAZR1R5K", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710590982291064, "job": 1, "event": "recovery_finished"} 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559d9ec26000 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: DB pointer 0x559d9ec10000 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:09:42.470 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 98.49 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 98.49 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:09:42.471 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 90.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T12:09:42.472 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 12.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 12.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x559d9eb33090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???) e1 preinit fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).mds e1 new map 2024-03-16T12:09:42.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).mds e1 print_map 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:09:42.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 systemd[1]: Started Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: monmap epoch 1 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: last_changed 2024-03-16T12:09:20.978440+0000 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: min_mon_release 19 (squid) 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: election_strategy: 1 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: fsmap 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T12:09:42.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:42 smithi098 ceph-mon[31633]: mgrmap e1: no daemons active 2024-03-16T12:09:43.694 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.942+0000 7f414a200700 1 Processor -- start 2024-03-16T12:09:43.694 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.942+0000 7f414a200700 1 -- start start 2024-03-16T12:09:43.694 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.943+0000 7f414a200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:43.694 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.943+0000 7f414a200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f414407be50 con 0x7f4144108680 2024-03-16T12:09:43.694 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.943+0000 7f4143400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.944+0000 7f4143400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49802/0 (socket says 172.21.15.98:49802) 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.944+0000 7f4143400700 1 -- 172.21.15.98:0/3084633853 learned_addr learned my addr 172.21.15.98:0/3084633853 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.944+0000 7f4143400700 1 -- 172.21.15.98:0/3084633853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414410afb0 con 0x7f4144108680 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.945+0000 7f4143400700 1 --2- 172.21.15.98:0/3084633853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f41340098a0 tx=0x7f4134009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88c6b253371495 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.945+0000 7f4142000700 1 -- 172.21.15.98:0/3084633853 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134004160 con 0x7f4144108680 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.945+0000 7f4142000700 1 -- 172.21.15.98:0/3084633853 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4134004300 con 0x7f4144108680 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.945+0000 7f4142000700 1 -- 172.21.15.98:0/3084633853 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134011b50 con 0x7f4144108680 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 -- 172.21.15.98:0/3084633853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 msgr2=0x7f414410aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 --2- 172.21.15.98:0/3084633853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f41340098a0 tx=0x7f4134009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 -- 172.21.15.98:0/3084633853 shutdown_connections 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 --2- 172.21.15.98:0/3084633853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414410aa70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 -- 172.21.15.98:0/3084633853 >> 172.21.15.98:0/3084633853 conn(0x7f4144102090 msgr2=0x7f41441044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:43.695 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 -- 172.21.15.98:0/3084633853 shutdown_connections 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.946+0000 7f414a200700 1 -- 172.21.15.98:0/3084633853 wait complete. 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.947+0000 7f414a200700 1 Processor -- start 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f414a200700 1 -- start start 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f414a200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f414a200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f414410bf60 con 0x7f4144108680 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f4143400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f4143400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49804/0 (socket says 172.21.15.98:49804) 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.948+0000 7f4143400700 1 -- 172.21.15.98:0/702979797 learned_addr learned my addr 172.21.15.98:0/702979797 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f4143400700 1 -- 172.21.15.98:0/702979797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414419e2f0 con 0x7f4144108680 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f4143400700 1 --2- 172.21.15.98:0/702979797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4134009fb0 tx=0x7f4134011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41340038b0 con 0x7f4144108680 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4134011e50 con 0x7f4144108680 2024-03-16T12:09:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134019420 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.949+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414419e580 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.950+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441a1230 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.950+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f41340195c0 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.950+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f413402dcb0 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.951+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4144063900 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:42.954+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f413402d440 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.029+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f41441a18c0 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.033+0000 7f413be00700 1 -- 172.21.15.98:0/702979797 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f4134003a50 con 0x7f4144108680 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 msgr2=0x7f414419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 --2- 172.21.15.98:0/702979797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4134009fb0 tx=0x7f4134011bf0 comp rx=0 tx=0).stop 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 shutdown_connections 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 --2- 172.21.15.98:0/702979797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4144108680 0x7f414419ddb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 >> 172.21.15.98:0/702979797 conn(0x7f4144102090 msgr2=0x7f4144102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.034+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 shutdown_connections 2024-03-16T12:09:43.698 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:43.035+0000 7f414a200700 1 -- 172.21.15.98:0/702979797 wait complete. 2024-03-16T12:09:43.698 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T12:09:43.752 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:09:43.752 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-03-16T12:09:43.753 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T12:09:43.754 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T12:09:44.171 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x 2024-03-16T12:09:44.171 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T12:09:44.406 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715.target.wants/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@.service. 2024-03-16T12:09:44.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:44 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/702979797' entity='client.admin' 2024-03-16T12:09:44.753 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:44 smithi098 systemd[1]: Starting Ceph mgr.x for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:09:45.068 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:44 smithi098 podman[31858]: 2024-03-16 12:09:44.684972266 +0000 UTC m=+0.032131125 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:09:45.068 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:44 smithi098 podman[31858]: 2024-03-16 12:09:44.800403855 +0000 UTC m=+0.147562746 container create 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:09:45.068 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 podman[31858]: 2024-03-16 12:09:45.079102271 +0000 UTC m=+0.426261215 container init 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T12:09:45.160 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T12:09:45.160 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-16T12:09:45.174 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T12:09:45.175 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:09:45.175 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-16T12:09:45.175 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T12:09:45.192 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T12:09:45.192 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-16T12:09:45.207 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T12:09:45.207 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:09:45.207 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-16T12:09:45.208 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T12:09:45.208 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-03-16T12:09:45.208 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-03-16T12:09:45.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 podman[31858]: 2024-03-16 12:09:45.088569168 +0000 UTC m=+0.435728040 container start 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:45.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 bash[31858]: 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d 2024-03-16T12:09:45.343 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 systemd[1]: Started Ceph mgr.x for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:09:45.894 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:45.844+0000 7f2de9cfe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T12:09:46.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:45 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/572069877' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T12:09:46.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:45.977+0000 7f2de9cfe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T12:09:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:09:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:09:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T12:09:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T12:09:46.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T12:09:46.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T12:09:46.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T12:09:46.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T12:09:46.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T12:09:46.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T12:09:46.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T12:09:46.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T12:09:46.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T12:09:46.594 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T12:09:34.984378+0000", 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T12:09:46.595 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.797+0000 7f3dac600700 1 Processor -- start 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3dac600700 1 -- start start 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3dac600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3dac600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3da407e140 con 0x7f3da407dd20 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3da9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3da9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49830/0 (socket says 172.21.15.98:49830) 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.798+0000 7f3da9c00700 1 -- 172.21.15.98:0/1347445699 learned_addr learned my addr 172.21.15.98:0/1347445699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.799+0000 7f3da9c00700 1 -- 172.21.15.98:0/1347445699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da407c820 con 0x7f3da407dd20 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.799+0000 7f3da9c00700 1 --2- 172.21.15.98:0/1347445699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098a0 tx=0x7f3d98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a652ea0581bb2de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.799+0000 7f3da3e00700 1 -- 172.21.15.98:0/1347445699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d98004160 con 0x7f3da407dd20 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3da3e00700 1 -- 172.21.15.98:0/1347445699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d98004300 con 0x7f3da407dd20 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3da3e00700 1 -- 172.21.15.98:0/1347445699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d98011b50 con 0x7f3da407dd20 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 -- 172.21.15.98:0/1347445699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 msgr2=0x7f3da407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:46.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 --2- 172.21.15.98:0/1347445699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098a0 tx=0x7f3d98009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 -- 172.21.15.98:0/1347445699 shutdown_connections 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 --2- 172.21.15.98:0/1347445699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da407dd20 0x7f3da407c2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 -- 172.21.15.98:0/1347445699 >> 172.21.15.98:0/1347445699 conn(0x7f3da41032a0 msgr2=0x7f3da41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 -- 172.21.15.98:0/1347445699 shutdown_connections 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.800+0000 7f3dac600700 1 -- 172.21.15.98:0/1347445699 wait complete. 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.801+0000 7f3dac600700 1 Processor -- start 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3dac600700 1 -- start start 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3dac600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:46.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3dac600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3da410e040 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3da9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3da9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49834/0 (socket says 172.21.15.98:49834) 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.802+0000 7f3da9c00700 1 -- 172.21.15.98:0/572069877 learned_addr learned my addr 172.21.15.98:0/572069877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3da9c00700 1 -- 172.21.15.98:0/572069877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da419e8c0 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3da9c00700 1 --2- 172.21.15.98:0/572069877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d980119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d980038b0 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da419eb50 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d98003a50 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d9801b440 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.803+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da41a1630 con 0x7f3da419df60 2024-03-16T12:09:46.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.804+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3d9801ba80 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.804+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3d9802e730 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.805+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da4063900 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.808+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3d9802e9d0 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.886+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3da41a19a0 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.887+0000 7f3da2000700 1 -- 172.21.15.98:0/572069877 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3d98033030 con 0x7f3da419df60 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.888+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 msgr2=0x7f3da419e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.888+0000 7f3dac600700 1 --2- 172.21.15.98:0/572069877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d980119e0 comp rx=0 tx=0).stop 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.889+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 shutdown_connections 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.889+0000 7f3dac600700 1 --2- 172.21.15.98:0/572069877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da419df60 0x7f3da419e380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.889+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 >> 172.21.15.98:0/572069877 conn(0x7f3da41032a0 msgr2=0x7f3da4103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:46.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.889+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 shutdown_connections 2024-03-16T12:09:46.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:45.889+0000 7f3dac600700 1 -- 172.21.15.98:0/572069877 wait complete. 2024-03-16T12:09:46.600 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-03-16T12:09:46.899 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:46.791+0000 7f2de9cfe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T12:09:47.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:46.910+0000 7f2de9cfe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T12:09:47.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.092+0000 7f2de9cfe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T12:09:47.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.214+0000 7f2de9cfe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T12:09:47.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.325+0000 7f2de9cfe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T12:09:48.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.653+0000 7f2de9cfe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T12:09:48.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.791+0000 7f2de9cfe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T12:09:48.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:47.901+0000 7f2de9cfe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T12:09:48.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:48.024+0000 7f2de9cfe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T12:09:48.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:48.239+0000 7f2de9cfe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T12:09:49.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:49 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/773203541' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T12:09:50.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:49.554+0000 7f2de9cfe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T12:09:50.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:49.739+0000 7f2de9cfe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T12:09:50.054 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:09:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:09:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T12:09:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T12:09:50.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T12:09:50.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T12:09:50.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T12:09:50.057 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-16T12:09:50.058 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T12:09:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T12:09:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T12:09:50.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T12:09:50.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T12:09:50.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:50.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T12:09:50.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T12:09:50.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:50.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T12:09:34.984378+0000", 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.193+0000 7f6379a00700 1 Processor -- start 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.193+0000 7f6379a00700 1 -- start start 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.193+0000 7f6379a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.194+0000 7f6379a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f637407c8b0 con 0x7f637407de80 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.194+0000 7f6373e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.194+0000 7f6373e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50248/0 (socket says 172.21.15.98:50248) 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.194+0000 7f6373e00700 1 -- 172.21.15.98:0/4256049304 learned_addr learned my addr 172.21.15.98:0/4256049304 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.195+0000 7f6373e00700 1 -- 172.21.15.98:0/4256049304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637407ca30 con 0x7f637407de80 2024-03-16T12:09:50.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.195+0000 7f6373e00700 1 --2- 172.21.15.98:0/4256049304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63640098a0 tx=0x7f6364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec5380410b05645 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:50.070 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.195+0000 7f6372a00700 1 -- 172.21.15.98:0/4256049304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6364004160 con 0x7f637407de80 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.195+0000 7f6372a00700 1 -- 172.21.15.98:0/4256049304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6364004300 con 0x7f637407de80 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.195+0000 7f6372a00700 1 -- 172.21.15.98:0/4256049304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6364011b50 con 0x7f637407de80 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 -- 172.21.15.98:0/4256049304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 msgr2=0x7f637407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 --2- 172.21.15.98:0/4256049304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63640098a0 tx=0x7f6364009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 -- 172.21.15.98:0/4256049304 shutdown_connections 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 --2- 172.21.15.98:0/4256049304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637407c2e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 -- 172.21.15.98:0/4256049304 >> 172.21.15.98:0/4256049304 conn(0x7f6374102f70 msgr2=0x7f6374105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 -- 172.21.15.98:0/4256049304 shutdown_connections 2024-03-16T12:09:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.196+0000 7f6379a00700 1 -- 172.21.15.98:0/4256049304 wait complete. 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.197+0000 7f6379a00700 1 Processor -- start 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.197+0000 7f6379a00700 1 -- start start 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6379a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6379a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f637410df60 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6373e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6373e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50258/0 (socket says 172.21.15.98:50258) 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6373e00700 1 -- 172.21.15.98:0/773203541 learned_addr learned my addr 172.21.15.98:0/773203541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.198+0000 7f6373e00700 1 -- 172.21.15.98:0/773203541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637419e080 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.199+0000 7f6373e00700 1 --2- 172.21.15.98:0/773203541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6364000c00 tx=0x7f6364011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.199+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6364003b20 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.199+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6364003cc0 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.199+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637419e310 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.199+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6364024910 con 0x7f637407de80 2024-03-16T12:09:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.200+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6364022020 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.200+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63741a1000 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.200+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6364011e00 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.201+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6374063900 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.203+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6364033080 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.283+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f63741a15a0 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.284+0000 7f6370c00700 1 -- 172.21.15.98:0/773203541 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f636401acd0 con 0x7f637407de80 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 msgr2=0x7f637419db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 --2- 172.21.15.98:0/773203541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6364000c00 tx=0x7f6364011c10 comp rx=0 tx=0).stop 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 shutdown_connections 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 --2- 172.21.15.98:0/773203541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f637407de80 0x7f637419db40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 >> 172.21.15.98:0/773203541 conn(0x7f6374102f70 msgr2=0x7f6374103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 shutdown_connections 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:49.285+0000 7f6379a00700 1 -- 172.21.15.98:0/773203541 wait complete. 2024-03-16T12:09:50.076 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-03-16T12:09:50.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:50.156+0000 7f2de9cfe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T12:09:50.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:50.273+0000 7f2de9cfe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T12:09:50.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:50.525+0000 7f2de9cfe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:09:51.000 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:50.764+0000 7f2de9cfe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T12:09:51.426 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:51.126+0000 7f2de9cfe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:09:51.726 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:51.437+0000 7f2de9cfe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T12:09:51.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:51.736+0000 7f2de9cfe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T12:09:53.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:52 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2594509560' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T12:09:53.153 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:52.781+0000 7f2de9cfe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T12:09:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:09:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:09:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T12:09:53.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T12:09:53.457 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T12:09:53.465 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T12:09:53.465 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T12:09:34.984378+0000", 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.659+0000 7f4bd4200700 1 Processor -- start 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.659+0000 7f4bd4200700 1 -- start start 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.659+0000 7f4bd4200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.659+0000 7f4bd4200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc07e100 con 0x7f4bcc07dce0 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.660+0000 7f4bd1800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.660+0000 7f4bd1800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50272/0 (socket says 172.21.15.98:50272) 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.660+0000 7f4bd1800700 1 -- 172.21.15.98:0/3982782393 learned_addr learned my addr 172.21.15.98:0/3982782393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:53.468 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.660+0000 7f4bd1800700 1 -- 172.21.15.98:0/3982782393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc07c7e0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.660+0000 7f4bd1800700 1 --2- 172.21.15.98:0/3982782393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00098a0 tx=0x7f4bc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d184df448eabfeb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.661+0000 7f4bcbe00700 1 -- 172.21.15.98:0/3982782393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0004160 con 0x7f4bcc07dce0 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.661+0000 7f4bcbe00700 1 -- 172.21.15.98:0/3982782393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bc0004300 con 0x7f4bcc07dce0 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.661+0000 7f4bcbe00700 1 -- 172.21.15.98:0/3982782393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0011b50 con 0x7f4bcc07dce0 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.661+0000 7f4bd4200700 1 -- 172.21.15.98:0/3982782393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 msgr2=0x7f4bcc07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.661+0000 7f4bd4200700 1 --2- 172.21.15.98:0/3982782393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00098a0 tx=0x7f4bc0009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.662+0000 7f4bd4200700 1 -- 172.21.15.98:0/3982782393 shutdown_connections 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.662+0000 7f4bd4200700 1 --2- 172.21.15.98:0/3982782393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc07c2a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.662+0000 7f4bd4200700 1 -- 172.21.15.98:0/3982782393 >> 172.21.15.98:0/3982782393 conn(0x7f4bcc1032a0 msgr2=0x7f4bcc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.662+0000 7f4bd4200700 1 -- 172.21.15.98:0/3982782393 shutdown_connections 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.662+0000 7f4bd4200700 1 -- 172.21.15.98:0/3982782393 wait complete. 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.663+0000 7f4bd4200700 1 Processor -- start 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.663+0000 7f4bd4200700 1 -- start start 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.663+0000 7f4bd4200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:53.469 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.663+0000 7f4bd4200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10e030 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.663+0000 7f4bd1800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bd1800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50276/0 (socket says 172.21.15.98:50276) 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bd1800700 1 -- 172.21.15.98:0/2594509560 learned_addr learned my addr 172.21.15.98:0/2594509560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bd1800700 1 -- 172.21.15.98:0/2594509560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc19e2e0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bd1800700 1 --2- 172.21.15.98:0/2594509560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0011b60 tx=0x7f4bc0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0003970 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bc0003b10 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc19e570 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.664+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0024610 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.665+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc1a1220 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.665+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4bc0024ca0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.665+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4bc0011e00 con 0x7f4bcc07dce0 2024-03-16T12:09:53.470 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.666+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bcc063900 con 0x7f4bcc07dce0 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.668+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4bc001a9f0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.748+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4bcc1a13f0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.749+0000 7f4bca000700 1 -- 172.21.15.98:0/2594509560 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f4bcc1a13f0 con 0x7f4bcc07dce0 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 msgr2=0x7f4bcc19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 --2- 172.21.15.98:0/2594509560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0011b60 tx=0x7f4bc0011c40 comp rx=0 tx=0).stop 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 shutdown_connections 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 --2- 172.21.15.98:0/2594509560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bcc07dce0 0x7f4bcc19dda0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 >> 172.21.15.98:0/2594509560 conn(0x7f4bcc1032a0 msgr2=0x7f4bcc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 shutdown_connections 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:52.750+0000 7f4bd4200700 1 -- 172.21.15.98:0/2594509560 wait complete. 2024-03-16T12:09:53.471 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (3/15)... 2024-03-16T12:09:53.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:53.497+0000 7f2de9cfe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T12:09:53.750 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:53.625+0000 7f2de9cfe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:09:53.900+0000 7f2de9cfe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: Activating manager daemon x 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: mgrmap e2: x(active, starting, since 0.00983717s) 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:09:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T12:09:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: Manager daemon x is now available 2024-03-16T12:09:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T12:09:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T12:09:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' 2024-03-16T12:09:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:53 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' 2024-03-16T12:09:55.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:54 smithi098 ceph-mon[31633]: from='mgr.14100 172.21.15.98:0/1079054016' entity='mgr.x' 2024-03-16T12:09:55.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:54 smithi098 ceph-mon[31633]: mgrmap e3: x(active, since 1.01971s) 2024-03-16T12:09:56.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:56 smithi098 ceph-mon[31633]: mgrmap e4: x(active, since 2s) 2024-03-16T12:09:56.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1120252221' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T12:09:57.009 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T12:09:57.022 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T12:09:57.023 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T12:09:57.024 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T12:09:34.984378+0000", 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.042+0000 7fd427c00700 1 Processor -- start 2024-03-16T12:09:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.042+0000 7fd427c00700 1 -- start start 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.042+0000 7fd427c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.042+0000 7fd427c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd42010a7c0 con 0x7fd420109dd0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.043+0000 7fd425200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.043+0000 7fd425200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50312/0 (socket says 172.21.15.98:50312) 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.043+0000 7fd425200700 1 -- 172.21.15.98:0/2235287887 learned_addr learned my addr 172.21.15.98:0/2235287887 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.043+0000 7fd425200700 1 -- 172.21.15.98:0/2235287887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42010afd0 con 0x7fd420109dd0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.043+0000 7fd425200700 1 --2- 172.21.15.98:0/2235287887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4100098a0 tx=0x7fd410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4c313e2b3ba36f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd41fe00700 1 -- 172.21.15.98:0/2235287887 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd410004160 con 0x7fd420109dd0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd41fe00700 1 -- 172.21.15.98:0/2235287887 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd410004300 con 0x7fd420109dd0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd41fe00700 1 -- 172.21.15.98:0/2235287887 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd410011b50 con 0x7fd420109dd0 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd427c00700 1 -- 172.21.15.98:0/2235287887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 msgr2=0x7fd42010a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd427c00700 1 --2- 172.21.15.98:0/2235287887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4100098a0 tx=0x7fd410009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:57.026 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd427c00700 1 -- 172.21.15.98:0/2235287887 shutdown_connections 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd427c00700 1 --2- 172.21.15.98:0/2235287887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42010a1f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.044+0000 7fd427c00700 1 -- 172.21.15.98:0/2235287887 >> 172.21.15.98:0/2235287887 conn(0x7fd42007d1d0 msgr2=0x7fd42007d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.045+0000 7fd427c00700 1 -- 172.21.15.98:0/2235287887 shutdown_connections 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.045+0000 7fd427c00700 1 -- 172.21.15.98:0/2235287887 wait complete. 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.046+0000 7fd427c00700 1 Processor -- start 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.046+0000 7fd427c00700 1 -- start start 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.046+0000 7fd427c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.046+0000 7fd427c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd42010e380 con 0x7fd420109dd0 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd425200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd425200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50322/0 (socket says 172.21.15.98:50322) 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd425200700 1 -- 172.21.15.98:0/1120252221 learned_addr learned my addr 172.21.15.98:0/1120252221 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd425200700 1 -- 172.21.15.98:0/1120252221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42019e300 con 0x7fd420109dd0 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd425200700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd410011850 tx=0x7fd410011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:57.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.047+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd410003af0 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.048+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42019e590 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.048+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd410020070 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.048+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4100195a0 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.048+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4201a1280 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.049+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fd4100198a0 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.049+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd420063900 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.049+0000 7fd41e000700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 0x7fd40403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.049+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd410011e00 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.049+0000 7fd417e00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 0x7fd40403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.050+0000 7fd417e00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 0x7fd40403b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd408006fd0 tx=0x7fd408006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.056+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd410003c90 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.357+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd4201a1450 con 0x7fd420109dd0 2024-03-16T12:09:57.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.357+0000 7fd41e000700 1 -- 172.21.15.98:0/1120252221 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd4201a1450 con 0x7fd420109dd0 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 msgr2=0x7fd40403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 0x7fd40403b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd408006fd0 tx=0x7fd408006e40 comp rx=0 tx=0).stop 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 msgr2=0x7fd42019ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd410011850 tx=0x7fd410011c10 comp rx=0 tx=0).stop 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 shutdown_connections 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7fd404038ed0 0x7fd40403b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 --2- 172.21.15.98:0/1120252221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd420109dd0 0x7fd42019ddc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.360+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 >> 172.21.15.98:0/1120252221 conn(0x7fd42007d1d0 msgr2=0x7fd420107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.361+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 shutdown_connections 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:56.361+0000 7fd427c00700 1 -- 172.21.15.98:0/1120252221 wait complete. 2024-03-16T12:09:57.029 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-03-16T12:09:58.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:57 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4241222614' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T12:09:58.583 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-03-16T12:09:58.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.651+0000 7f9a25a00700 1 Processor -- start 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.651+0000 7f9a25a00700 1 -- start start 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.652+0000 7f9a25a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.652+0000 7f9a25a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2007c840 con 0x7f9a2007dd40 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.652+0000 7f9a1ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.652+0000 7f9a1ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59510/0 (socket says 172.21.15.98:59510) 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.652+0000 7f9a1ea00700 1 -- 172.21.15.98:0/4096572696 learned_addr learned my addr 172.21.15.98:0/4096572696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.653+0000 7f9a1ea00700 1 -- 172.21.15.98:0/4096572696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2007c9c0 con 0x7f9a2007dd40 2024-03-16T12:09:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.653+0000 7f9a1ea00700 1 --2- 172.21.15.98:0/4096572696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098a0 tx=0x7f9a10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a2e0cc521ea8e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.653+0000 7f9a1d600700 1 -- 172.21.15.98:0/4096572696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a10004160 con 0x7f9a2007dd40 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a1d600700 1 -- 172.21.15.98:0/4096572696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a10004300 con 0x7f9a2007dd40 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a1d600700 1 -- 172.21.15.98:0/4096572696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a10011b50 con 0x7f9a2007dd40 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 -- 172.21.15.98:0/4096572696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 msgr2=0x7f9a2007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4096572696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098a0 tx=0x7f9a10009bb0 comp rx=0 tx=0).stop 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 -- 172.21.15.98:0/4096572696 shutdown_connections 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4096572696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2007c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 -- 172.21.15.98:0/4096572696 >> 172.21.15.98:0/4096572696 conn(0x7f9a201032a0 msgr2=0x7f9a201056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 -- 172.21.15.98:0/4096572696 shutdown_connections 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.654+0000 7f9a25a00700 1 -- 172.21.15.98:0/4096572696 wait complete. 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.655+0000 7f9a25a00700 1 Processor -- start 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a25a00700 1 -- start start 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a25a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:58.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a25a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010e030 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a1ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a1ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59526/0 (socket says 172.21.15.98:59526) 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a1ea00700 1 -- 172.21.15.98:0/4241222614 learned_addr learned my addr 172.21.15.98:0/4241222614 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.656+0000 7f9a1ea00700 1 -- 172.21.15.98:0/4241222614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2019e2e0 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a1ea00700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9a10011b60 tx=0x7f9a10011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a10003970 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a10003b10 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2019e570 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a10024610 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.657+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a201a1220 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.658+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f9a1001a9f0 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.658+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a20063900 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.659+0000 7f9a17400700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 0x7f9a0403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.659+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9a10011e00 con 0x7f9a2007dd40 2024-03-16T12:09:58.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.659+0000 7f9a1e000700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 0x7f9a0403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.660+0000 7f9a1e000700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 0x7f9a0403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c006fd0 tx=0x7f9a0c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.664+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a10019e10 con 0x7f9a2007dd40 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.884+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f9a201a1440 con 0x7f9a2007dd40 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.886+0000 7f9a17400700 1 -- 172.21.15.98:0/4241222614 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f9a201a1440 con 0x7f9a2007dd40 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 msgr2=0x7f9a0403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 0x7f9a0403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c006fd0 tx=0x7f9a0c006e40 comp rx=0 tx=0).stop 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 msgr2=0x7f9a2019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9a10011b60 tx=0x7f9a10011c40 comp rx=0 tx=0).stop 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 shutdown_connections 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f9a04038f20 0x7f9a0403b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 --2- 172.21.15.98:0/4241222614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2007dd40 0x7f9a2019dda0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.889+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 >> 172.21.15.98:0/4241222614 conn(0x7f9a201032a0 msgr2=0x7f9a20103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.890+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 shutdown_connections 2024-03-16T12:09:58.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:57.890+0000 7f9a25a00700 1 -- 172.21.15.98:0/4241222614 wait complete. 2024-03-16T12:09:58.589 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-03-16T12:09:59.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:09:59 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1596857805' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T12:09:59.750 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:59 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: ignoring --setuser ceph since I am not root 2024-03-16T12:09:59.750 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:09:59 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: ignoring --setgroup ceph since I am not root 2024-03-16T12:10:00.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.212+0000 7ff422e00700 1 Processor -- start 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.212+0000 7ff422e00700 1 -- start start 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.212+0000 7ff422e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.212+0000 7ff422e00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c07be10 con 0x7ff41c1086d0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.213+0000 7ff41be00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.213+0000 7ff41be00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59530/0 (socket says 172.21.15.98:59530) 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.213+0000 7ff41be00700 1 -- 172.21.15.98:0/1791459868 learned_addr learned my addr 172.21.15.98:0/1791459868 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.213+0000 7ff41be00700 1 -- 172.21.15.98:0/1791459868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c10b000 con 0x7ff41c1086d0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.214+0000 7ff41be00700 1 --2- 172.21.15.98:0/1791459868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009e80 tx=0x7ff40c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff426fd4c87b244 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.214+0000 7ff41b400700 1 -- 172.21.15.98:0/1791459868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff40c017070 con 0x7ff41c1086d0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.214+0000 7ff41b400700 1 -- 172.21.15.98:0/1791459868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff40c00b8c0 con 0x7ff41c1086d0 2024-03-16T12:10:00.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.214+0000 7ff41b400700 1 -- 172.21.15.98:0/1791459868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff40c0045c0 con 0x7ff41c1086d0 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 -- 172.21.15.98:0/1791459868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 msgr2=0x7ff41c10aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 --2- 172.21.15.98:0/1791459868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009e80 tx=0x7ff40c00b330 comp rx=0 tx=0).stop 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 -- 172.21.15.98:0/1791459868 shutdown_connections 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 --2- 172.21.15.98:0/1791459868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c10aac0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 -- 172.21.15.98:0/1791459868 >> 172.21.15.98:0/1791459868 conn(0x7ff41c102100 msgr2=0x7ff41c104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 -- 172.21.15.98:0/1791459868 shutdown_connections 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.215+0000 7ff422e00700 1 -- 172.21.15.98:0/1791459868 wait complete. 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.217+0000 7ff422e00700 1 Processor -- start 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.217+0000 7ff422e00700 1 -- start start 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.217+0000 7ff422e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:00.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.217+0000 7ff422e00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c10bf50 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.217+0000 7ff41be00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.218+0000 7ff41be00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59546/0 (socket says 172.21.15.98:59546) 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.218+0000 7ff41be00700 1 -- 172.21.15.98:0/1596857805 learned_addr learned my addr 172.21.15.98:0/1596857805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.218+0000 7ff41be00700 1 -- 172.21.15.98:0/1596857805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c19c050 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.218+0000 7ff41be00700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009530 tx=0x7ff40c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.219+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff40c020020 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.219+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff40c003b50 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.219+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41c19c2e0 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.219+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff40c017040 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.219+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff41c19efd0 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.220+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7ff40c023020 con 0x7ff41c1086d0 2024-03-16T12:10:00.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.220+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41c063900 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.220+0000 7ff419600700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 0x7ff40003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.220+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff40c00f050 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.221+0000 7ff413400700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 0x7ff40003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.221+0000 7ff413400700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 0x7ff40003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff404006fd0 tx=0x7ff404006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.228+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff40c02a020 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.508+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff41c19f240 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.561+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff40c04c890 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.564+0000 7ff419600700 1 -- 172.21.15.98:0/1596857805 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7ff40c011c90 con 0x7ff41c1086d0 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 msgr2=0x7ff40003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:00.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 0x7ff40003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff404006fd0 tx=0x7ff404006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 msgr2=0x7ff41c19bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009530 tx=0x7ff40c00bd50 comp rx=0 tx=0).stop 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 shutdown_connections 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7ff400038f20 0x7ff40003b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 --2- 172.21.15.98:0/1596857805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff41c1086d0 0x7ff41c19bb10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.565+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 >> 172.21.15.98:0/1596857805 conn(0x7ff41c102100 msgr2=0x7ff41c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.566+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 shutdown_connections 2024-03-16T12:10:00.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:09:59.567+0000 7ff422e00700 1 -- 172.21.15.98:0/1596857805 wait complete. 2024-03-16T12:10:00.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:00 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:00.346+0000 7f98b5bc7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T12:10:00.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:00 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:00.473+0000 7f98b5bc7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T12:10:00.862 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:00 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1596857805' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T12:10:00.862 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:00 smithi098 ceph-mon[31633]: mgrmap e5: x(active, since 5s) 2024-03-16T12:10:01.408 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:01.297+0000 7f98b5bc7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T12:10:01.409 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:01.419+0000 7f98b5bc7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T12:10:01.713 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/351848301' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T12:10:01.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:01.600+0000 7f98b5bc7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T12:10:01.714 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:01.724+0000 7f98b5bc7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T12:10:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:10:01.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.872+0000 7f39c8000700 1 Processor -- start 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.873+0000 7f39c8000700 1 -- start start 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.873+0000 7f39c8000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.873+0000 7f39c8000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c007bdb0 con 0x7f39c0108750 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.874+0000 7f39c5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.874+0000 7f39c5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59558/0 (socket says 172.21.15.98:59558) 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.874+0000 7f39c5600700 1 -- 172.21.15.98:0/1298312116 learned_addr learned my addr 172.21.15.98:0/1298312116 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.874+0000 7f39c5600700 1 -- 172.21.15.98:0/1298312116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c010b080 con 0x7f39c0108750 2024-03-16T12:10:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.875+0000 7f39c5600700 1 --2- 172.21.15.98:0/1298312116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f39b40098a0 tx=0x7f39b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cb678df8cbecce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.875+0000 7f39bfe00700 1 -- 172.21.15.98:0/1298312116 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39b4004160 con 0x7f39c0108750 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.875+0000 7f39bfe00700 1 -- 172.21.15.98:0/1298312116 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39b4004300 con 0x7f39c0108750 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.875+0000 7f39bfe00700 1 -- 172.21.15.98:0/1298312116 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39b4011b50 con 0x7f39c0108750 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 -- 172.21.15.98:0/1298312116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 msgr2=0x7f39c010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 --2- 172.21.15.98:0/1298312116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f39b40098a0 tx=0x7f39b4009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 -- 172.21.15.98:0/1298312116 shutdown_connections 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 --2- 172.21.15.98:0/1298312116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c010ab40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 -- 172.21.15.98:0/1298312116 >> 172.21.15.98:0/1298312116 conn(0x7f39c01020c0 msgr2=0x7f39c01044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:01.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 -- 172.21.15.98:0/1298312116 shutdown_connections 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.876+0000 7f39c8000700 1 -- 172.21.15.98:0/1298312116 wait complete. 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.877+0000 7f39c8000700 1 Processor -- start 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.877+0000 7f39c8000700 1 -- start start 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c8000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c8000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c010bf70 con 0x7f39c0108750 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59564/0 (socket says 172.21.15.98:59564) 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c5600700 1 -- 172.21.15.98:0/351848301 learned_addr learned my addr 172.21.15.98:0/351848301 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.878+0000 7f39c5600700 1 -- 172.21.15.98:0/351848301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c019e250 con 0x7f39c0108750 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39c5600700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39b4011b60 tx=0x7f39b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39b4003b20 con 0x7f39c0108750 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39b4003cc0 con 0x7f39c0108750 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c019e4e0 con 0x7f39c0108750 2024-03-16T12:10:01.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39b4024840 con 0x7f39c0108750 2024-03-16T12:10:01.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.879+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c01a11c0 con 0x7f39c0108750 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.880+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c0063900 con 0x7f39c0108750 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.880+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f39b4022020 con 0x7f39c0108750 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.881+0000 7f39be000700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 0x7f39a803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.881+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f39b4011e00 con 0x7f39c0108750 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.881+0000 7f39c4c00700 1 -- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 msgr2=0x7f39a803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.881+0000 7f39c4c00700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 0x7f39a803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:10:01.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:00.886+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f39b404caa0 con 0x7f39c0108750 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.082+0000 7f39c4c00700 1 -- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 msgr2=0x7f39a803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.082+0000 7f39c4c00700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 0x7f39a803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.165+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f39c01a1390 con 0x7f39c0108750 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.166+0000 7f39be000700 1 -- 172.21.15.98:0/351848301 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f39c01a1390 con 0x7f39c0108750 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 msgr2=0x7f39a803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 0x7f39a803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 msgr2=0x7f39c019dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39b4011b60 tx=0x7f39b4011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 shutdown_connections 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f39a8038f70 0x7f39a803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 --2- 172.21.15.98:0/351848301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108750 0x7f39c019dd10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.169+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 >> 172.21.15.98:0/351848301 conn(0x7f39c01020c0 msgr2=0x7f39c0102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.170+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 shutdown_connections 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:01.170+0000 7f39c8000700 1 -- 172.21.15.98:0/351848301 wait complete. 2024-03-16T12:10:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-03-16T12:10:01.896 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-03-16T12:10:01.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:01.833+0000 7f98b5bc7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T12:10:02.308 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:02.173+0000 7f98b5bc7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T12:10:02.731 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:02.319+0000 7f98b5bc7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T12:10:02.731 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:02.431+0000 7f98b5bc7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T12:10:02.731 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:02.560+0000 7f98b5bc7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T12:10:02.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:02.777+0000 7f98b5bc7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T12:10:04.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:04.024+0000 7f98b5bc7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T12:10:04.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:04.194+0000 7f98b5bc7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T12:10:04.962 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:04.605+0000 7f98b5bc7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T12:10:05.033 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:04.721+0000 7f98b5bc7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T12:10:05.033 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:04.973+0000 7f98b5bc7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:10:05.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:05 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:05.215+0000 7f98b5bc7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T12:10:05.886 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:05 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:05.583+0000 7f98b5bc7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:10:05.886 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:05 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:05.897+0000 7f98b5bc7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T12:10:06.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:06 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:06.195+0000 7f98b5bc7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T12:10:07.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:07.195+0000 7f98b5bc7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T12:10:08.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:07.878+0000 7f98b5bc7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T12:10:08.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:08.005+0000 7f98b5bc7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T12:10:08.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:08.282+0000 7f98b5bc7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T12:10:08.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: Active manager daemon x restarted 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: Activating manager daemon x 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: mgrmap e6: x(active, starting, since 0.0103618s) 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: Manager daemon x is now available 2024-03-16T12:10:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:08 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:09.638 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: Found migration_current of "None". Setting to last migration. 2024-03-16T12:10:09.638 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:09.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:09 smithi098 ceph-mon[31633]: mgrmap e7: x(active, since 1.02056s) 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.533+0000 7f22d2600700 1 Processor -- start 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.533+0000 7f22d2600700 1 -- start start 2024-03-16T12:10:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.533+0000 7f22d2600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.533+0000 7f22d2600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc07be30 con 0x7f22cc108690 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.534+0000 7f22cb400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.534+0000 7f22cb400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59566/0 (socket says 172.21.15.98:59566) 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.534+0000 7f22cb400700 1 -- 172.21.15.98:0/1483300913 learned_addr learned my addr 172.21.15.98:0/1483300913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.534+0000 7f22cb400700 1 -- 172.21.15.98:0/1483300913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc10afc0 con 0x7f22cc108690 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.534+0000 7f22cb400700 1 --2- 172.21.15.98:0/1483300913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0098a0 tx=0x7f22bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0ef52e37cccd26 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.535+0000 7f22ca000700 1 -- 172.21.15.98:0/1483300913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc004160 con 0x7f22cc108690 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.535+0000 7f22ca000700 1 -- 172.21.15.98:0/1483300913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22bc004300 con 0x7f22cc108690 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.535+0000 7f22ca000700 1 -- 172.21.15.98:0/1483300913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc011b50 con 0x7f22cc108690 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.535+0000 7f22d2600700 1 -- 172.21.15.98:0/1483300913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 msgr2=0x7f22cc10aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:09.992 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.535+0000 7f22d2600700 1 --2- 172.21.15.98:0/1483300913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0098a0 tx=0x7f22bc009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.536+0000 7f22d2600700 1 -- 172.21.15.98:0/1483300913 shutdown_connections 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.536+0000 7f22d2600700 1 --2- 172.21.15.98:0/1483300913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc10aa80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.536+0000 7f22d2600700 1 -- 172.21.15.98:0/1483300913 >> 172.21.15.98:0/1483300913 conn(0x7f22cc1020a0 msgr2=0x7f22cc1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.536+0000 7f22d2600700 1 -- 172.21.15.98:0/1483300913 shutdown_connections 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.536+0000 7f22d2600700 1 -- 172.21.15.98:0/1483300913 wait complete. 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.537+0000 7f22d2600700 1 Processor -- start 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.537+0000 7f22d2600700 1 -- start start 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.537+0000 7f22d2600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.537+0000 7f22d2600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc10bf20 con 0x7f22cc108690 2024-03-16T12:10:09.993 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.537+0000 7f22cb400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22cb400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59574/0 (socket says 172.21.15.98:59574) 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22cb400700 1 -- 172.21.15.98:0/1125166822 learned_addr learned my addr 172.21.15.98:0/1125166822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22cb400700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc19e2a0 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22cb400700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f22bc011b60 tx=0x7f22bc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc003970 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22bc003b10 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc19e530 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.538+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc024610 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.539+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc1a1220 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.540+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f22bc01a9f0 con 0x7f22cc108690 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.540+0000 7f22c3e00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.540+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22b003baf0 con 0x7f22b0038f20 2024-03-16T12:10:09.994 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.540+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f22bc011e00 con 0x7f22cc108690 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.541+0000 7f22caa00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.541+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.742+0000 7f22caa00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:02.742+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:03.143+0000 7f22caa00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:03.143+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:03.944+0000 7f22caa00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:03.944+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:05.546+0000 7f22caa00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3029136477 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:05.546+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:08.293+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7f22bc024c90 con 0x7f22cc108690 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:08.293+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 msgr2=0x7f22b003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:08.293+0000 7f22c3e00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.995 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.301+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f22bc02c8a0 con 0x7f22cc108690 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.301+0000 7f22c3e00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 0x7f22b003ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.301+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22bc006880 con 0x7f22b003c810 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.302+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 0x7f22b003ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.304+0000 7f22caa00700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 0x7f22b003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f22b8003a10 tx=0x7f22b8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.305+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f22bc006880 con 0x7f22b003c810 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.317+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f22cc063900 con 0x7f22b003c810 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.317+0000 7f22c3e00700 1 -- 172.21.15.98:0/1125166822 <== mgr.14120 v2:172.21.15.98:6800/3396936170 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f22cc063900 con 0x7f22b003c810 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 msgr2=0x7f22b003ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 0x7f22b003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f22b8003a10 tx=0x7f22b8009290 comp rx=0 tx=0).stop 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 msgr2=0x7f22cc19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f22bc011b60 tx=0x7f22bc011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 shutdown_connections 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f22b003c810 0x7f22b003ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22cc108690 0x7f22cc19dd60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.996 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 --2- 172.21.15.98:0/1125166822 >> [v2:172.21.15.98:6800/3029136477,v1:172.21.15.98:6801/3029136477] conn(0x7f22b0038f20 0x7f22b003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:09.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.318+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 >> 172.21.15.98:0/1125166822 conn(0x7f22cc1020a0 msgr2=0x7f22cc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:09.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.319+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 shutdown_connections 2024-03-16T12:10:09.997 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:09.319+0000 7f22d2600700 1 -- 172.21.15.98:0/1125166822 wait complete. 2024-03-16T12:10:09.997 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-03-16T12:10:09.997 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-03-16T12:10:10.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:10 smithi098 ceph-mon[31633]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T12:10:10.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:10 smithi098 ceph-mon[31633]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T12:10:10.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:10 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:11.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:09] ENGINE Bus STARTING 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:09] ENGINE Serving on https://172.21.15.98:7150 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:10] ENGINE Serving on http://172.21.15.98:8765 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:10] ENGINE Bus STARTED 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: mgrmap e8: x(active, since 2s) 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:11.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:11 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.705+0000 7f4a28800700 1 Processor -- start 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.705+0000 7f4a28800700 1 -- start start 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a28800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a28800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a201085a0 con 0x7f4a20107c40 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a25e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a25e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47054/0 (socket says 172.21.15.98:47054) 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a25e00700 1 -- 172.21.15.98:0/2619583782 learned_addr learned my addr 172.21.15.98:0/2619583782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.706+0000 7f4a25e00700 1 -- 172.21.15.98:0/2619583782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a20108720 con 0x7f4a20107c40 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.707+0000 7f4a25e00700 1 --2- 172.21.15.98:0/2619583782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098a0 tx=0x7f4a14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f494ae3ecb62ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:11.781 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.707+0000 7f4a24a00700 1 -- 172.21.15.98:0/2619583782 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a14004160 con 0x7f4a20107c40 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.707+0000 7f4a24a00700 1 -- 172.21.15.98:0/2619583782 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4a14004300 con 0x7f4a20107c40 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.707+0000 7f4a24a00700 1 -- 172.21.15.98:0/2619583782 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a14011b50 con 0x7f4a20107c40 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 -- 172.21.15.98:0/2619583782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 msgr2=0x7f4a20108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 --2- 172.21.15.98:0/2619583782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098a0 tx=0x7f4a14009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 -- 172.21.15.98:0/2619583782 shutdown_connections 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 --2- 172.21.15.98:0/2619583782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a20108060 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 -- 172.21.15.98:0/2619583782 >> 172.21.15.98:0/2619583782 conn(0x7f4a201032a0 msgr2=0x7f4a201056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 -- 172.21.15.98:0/2619583782 shutdown_connections 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.708+0000 7f4a28800700 1 -- 172.21.15.98:0/2619583782 wait complete. 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.709+0000 7f4a28800700 1 Processor -- start 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.709+0000 7f4a28800700 1 -- start start 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a28800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a28800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a20109be0 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a25e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a25e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47068/0 (socket says 172.21.15.98:47068) 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a25e00700 1 -- 172.21.15.98:0/3274371919 learned_addr learned my addr 172.21.15.98:0/3274371919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.710+0000 7f4a25e00700 1 -- 172.21.15.98:0/3274371919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2019c060 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a25e00700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a14011b60 tx=0x7f4a14011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a14003970 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4a14003b10 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2019c2f0 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a14024610 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.711+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2019efe0 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.712+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f4a1401a9f0 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.712+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a20063900 con 0x7f4a20107c40 2024-03-16T12:10:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.713+0000 7f4a1aa00700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 0x7f4a0803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.713+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4a14011e00 con 0x7f4a20107c40 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.713+0000 7f4a25400700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 0x7f4a0803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.714+0000 7f4a25400700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 0x7f4a0803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a10006fd0 tx=0x7f4a10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.719+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a14019dd0 con 0x7f4a20107c40 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.939+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4a20104530 con 0x7f4a08038f20 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.950+0000 7f4a1aa00700 1 -- 172.21.15.98:0/3274371919 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4a20104530 con 0x7f4a08038f20 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 msgr2=0x7f4a0803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 0x7f4a0803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4a10006fd0 tx=0x7f4a10006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 msgr2=0x7f4a2019bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a14011b60 tx=0x7f4a14011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 shutdown_connections 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f4a08038f20 0x7f4a0803b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 --2- 172.21.15.98:0/3274371919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a20107c40 0x7f4a2019bb20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:11.784 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 >> 172.21.15.98:0/3274371919 conn(0x7f4a201032a0 msgr2=0x7f4a20103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:11.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 shutdown_connections 2024-03-16T12:10:11.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:10.953+0000 7f4a28800700 1 -- 172.21.15.98:0/3274371919 wait complete. 2024-03-16T12:10:12.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:12 smithi098 ceph-mon[31633]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:13.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T12:10:13.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.348+0000 7fbbe1400700 1 Processor -- start 2024-03-16T12:10:13.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.349+0000 7fbbe1400700 1 -- start start 2024-03-16T12:10:13.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.349+0000 7fbbe1400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:13.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.349+0000 7fbbe1400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc1085a0 con 0x7fbbdc107c40 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.350+0000 7fbbda800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.350+0000 7fbbda800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47076/0 (socket says 172.21.15.98:47076) 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.350+0000 7fbbda800700 1 -- 172.21.15.98:0/343053427 learned_addr learned my addr 172.21.15.98:0/343053427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.350+0000 7fbbda800700 1 -- 172.21.15.98:0/343053427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc108720 con 0x7fbbdc107c40 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.351+0000 7fbbda800700 1 --2- 172.21.15.98:0/343053427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0098a0 tx=0x7fbbcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d4a21ac66e1cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.352+0000 7fbbd9400700 1 -- 172.21.15.98:0/343053427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbcc004160 con 0x7fbbdc107c40 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.352+0000 7fbbd9400700 1 -- 172.21.15.98:0/343053427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbbcc004300 con 0x7fbbdc107c40 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.352+0000 7fbbd9400700 1 -- 172.21.15.98:0/343053427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbcc011b50 con 0x7fbbdc107c40 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.353+0000 7fbbe1400700 1 -- 172.21.15.98:0/343053427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 msgr2=0x7fbbdc108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.353+0000 7fbbe1400700 1 --2- 172.21.15.98:0/343053427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0098a0 tx=0x7fbbcc009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:13.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.353+0000 7fbbe1400700 1 -- 172.21.15.98:0/343053427 shutdown_connections 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.353+0000 7fbbe1400700 1 --2- 172.21.15.98:0/343053427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc108060 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.353+0000 7fbbe1400700 1 -- 172.21.15.98:0/343053427 >> 172.21.15.98:0/343053427 conn(0x7fbbdc1032a0 msgr2=0x7fbbdc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.354+0000 7fbbe1400700 1 -- 172.21.15.98:0/343053427 shutdown_connections 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.354+0000 7fbbe1400700 1 -- 172.21.15.98:0/343053427 wait complete. 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbe1400700 1 Processor -- start 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbe1400700 1 -- start start 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbe1400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbe1400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc109be0 con 0x7fbbdc107c40 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbda800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbda800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47078/0 (socket says 172.21.15.98:47078) 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.355+0000 7fbbda800700 1 -- 172.21.15.98:0/1513575213 learned_addr learned my addr 172.21.15.98:0/1513575213 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbda800700 1 -- 172.21.15.98:0/1513575213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc19e280 con 0x7fbbdc107c40 2024-03-16T12:10:13.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbda800700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc011b60 tx=0x7fbbcc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbcc003970 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbbcc003b10 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbdc19e510 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.356+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbcc024610 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.357+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbdc1a1200 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.358+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fbbcc0248c0 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.358+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbdc063900 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.358+0000 7fbbd3400700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 0x7fbbc003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.358+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbbcc011e00 con 0x7fbbdc107c40 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.359+0000 7fbbd9e00700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 0x7fbbc003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.359+0000 7fbbd9e00700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 0x7fbbc003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006fd0 tx=0x7fbbc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.368+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbcc02d910 con 0x7fbbdc107c40 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.592+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fbbdc104480 con 0x7fbbc0038f70 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.593+0000 7fbbd3400700 1 -- 172.21.15.98:0/1513575213 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fbbdc104480 con 0x7fbbc0038f70 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.596+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 msgr2=0x7fbbc003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.596+0000 7fbbe1400700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 0x7fbbc003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006fd0 tx=0x7fbbc8006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.596+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 msgr2=0x7fbbdc19dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.596+0000 7fbbe1400700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc011b60 tx=0x7fbbcc011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 shutdown_connections 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fbbc0038f70 0x7fbbc003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 --2- 172.21.15.98:0/1513575213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbdc107c40 0x7fbbdc19dd40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 >> 172.21.15.98:0/1513575213 conn(0x7fbbdc1032a0 msgr2=0x7fbbdc103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 shutdown_connections 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:12.597+0000 7fbbe1400700 1 -- 172.21.15.98:0/1513575213 wait complete. 2024-03-16T12:10:13.309 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-03-16T12:10:13.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:13 smithi098 ceph-mon[31633]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:14.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: Generating public/private rsa key pair. 2024-03-16T12:10:14.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: Your identification has been saved in /tmp/tmpdykqj3u5/key. 2024-03-16T12:10:14.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: Your public key has been saved in /tmp/tmpdykqj3u5/key.pub. 2024-03-16T12:10:14.500 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: The key fingerprint is: 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: SHA256:AYHaA0q3fUWCuww6X0MWio9tNAxR99aTXGYtLzOaMxs ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: The key's randomart image is: 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: +---[RSA 3072]----+ 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | .o. o+o.. +. | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | o..oo..+.=. . | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: |..=++ ooo= o | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: |...Bo=.. ..+ . | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | * *.o S o + | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | + + = E | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | + . . = | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | . . | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: | | 2024-03-16T12:10:14.501 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: +----[SHA256]-----+ 2024-03-16T12:10:14.902 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.930+0000 7f166ec00700 1 Processor -- start 2024-03-16T12:10:14.903 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.930+0000 7f166ec00700 1 -- start start 2024-03-16T12:10:14.903 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.931+0000 7f166ec00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:14.903 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.931+0000 7f166ec00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1668108560 con 0x7f1668107c00 2024-03-16T12:10:14.903 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.931+0000 7f1667e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:14.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.931+0000 7f1667e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47080/0 (socket says 172.21.15.98:47080) 2024-03-16T12:10:14.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.931+0000 7f1667e00700 1 -- 172.21.15.98:0/2928880939 learned_addr learned my addr 172.21.15.98:0/2928880939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:14.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.932+0000 7f1667e00700 1 -- 172.21.15.98:0/2928880939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16681086e0 con 0x7f1668107c00 2024-03-16T12:10:14.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.932+0000 7f1667e00700 1 --2- 172.21.15.98:0/2928880939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16580098a0 tx=0x7f1658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff47d25b8b7b5f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.932+0000 7f1667400700 1 -- 172.21.15.98:0/2928880939 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1658004160 con 0x7f1668107c00 2024-03-16T12:10:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f1667400700 1 -- 172.21.15.98:0/2928880939 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1658004300 con 0x7f1668107c00 2024-03-16T12:10:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f1667400700 1 -- 172.21.15.98:0/2928880939 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1658011b50 con 0x7f1668107c00 2024-03-16T12:10:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 -- 172.21.15.98:0/2928880939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 msgr2=0x7f1668108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 --2- 172.21.15.98:0/2928880939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16580098a0 tx=0x7f1658009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 -- 172.21.15.98:0/2928880939 shutdown_connections 2024-03-16T12:10:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 --2- 172.21.15.98:0/2928880939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f1668108020 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 -- 172.21.15.98:0/2928880939 >> 172.21.15.98:0/2928880939 conn(0x7f1668103240 msgr2=0x7f1668105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 -- 172.21.15.98:0/2928880939 shutdown_connections 2024-03-16T12:10:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.933+0000 7f166ec00700 1 -- 172.21.15.98:0/2928880939 wait complete. 2024-03-16T12:10:14.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.934+0000 7f166ec00700 1 Processor -- start 2024-03-16T12:10:14.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f166ec00700 1 -- start start 2024-03-16T12:10:14.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f166ec00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:14.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f166ec00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1668109ba0 con 0x7f1668107c00 2024-03-16T12:10:14.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f1667e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:14.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f1667e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47088/0 (socket says 172.21.15.98:47088) 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f1667e00700 1 -- 172.21.15.98:0/3693136102 learned_addr learned my addr 172.21.15.98:0/3693136102 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.935+0000 7f1667e00700 1 -- 172.21.15.98:0/3693136102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166819c040 con 0x7f1668107c00 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f1667e00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1658011b60 tx=0x7f1658011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1658003970 con 0x7f1668107c00 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1658003b10 con 0x7f1668107c00 2024-03-16T12:10:14.910 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1658024610 con 0x7f1668107c00 2024-03-16T12:10:14.911 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166819c2d0 con 0x7f1668107c00 2024-03-16T12:10:14.911 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.936+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166819efc0 con 0x7f1668107c00 2024-03-16T12:10:14.911 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.938+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f165801a9f0 con 0x7f1668107c00 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.937+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1668063900 con 0x7f1668107c00 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.938+0000 7f1665600700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 0x7f164c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.938+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1658011e00 con 0x7f1668107c00 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.938+0000 7f165fe00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 0x7f164c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.939+0000 7f165fe00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 0x7f164c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1650006fd0 tx=0x7f1650006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:13.943+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f165800b3e0 con 0x7f1668107c00 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.166+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f1668104550 con 0x7f164c038f70 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.272+0000 7f1665600700 1 -- 172.21.15.98:0/3693136102 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1668104550 con 0x7f164c038f70 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 msgr2=0x7f164c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 0x7f164c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1650006fd0 tx=0x7f1650006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 msgr2=0x7f166819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1658011b60 tx=0x7f1658011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 shutdown_connections 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f164c038f70 0x7f164c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 --2- 172.21.15.98:0/3693136102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1668107c00 0x7f166819bb00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 >> 172.21.15.98:0/3693136102 conn(0x7f1668103240 msgr2=0x7f1668103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.275+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 shutdown_connections 2024-03-16T12:10:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:14.276+0000 7f166ec00700 1 -- 172.21.15.98:0/3693136102 wait complete. 2024-03-16T12:10:15.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:15 smithi098 ceph-mon[31633]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:15.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:15 smithi098 ceph-mon[31633]: Generating ssh key... 2024-03-16T12:10:15.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:15 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:15 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:10:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.466+0000 7f096fe00700 1 Processor -- start 2024-03-16T12:10:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.466+0000 7f096fe00700 1 -- start start 2024-03-16T12:10:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.467+0000 7f096fe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.467+0000 7f096fe00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f096807c860 con 0x7f096807ddb0 2024-03-16T12:10:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.467+0000 7f096d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.467+0000 7f096d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47098/0 (socket says 172.21.15.98:47098) 2024-03-16T12:10:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.467+0000 7f096d400700 1 -- 172.21.15.98:0/1419289469 learned_addr learned my addr 172.21.15.98:0/1419289469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.468+0000 7f096d400700 1 -- 172.21.15.98:0/1419289469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096807c9e0 con 0x7f096807ddb0 2024-03-16T12:10:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.468+0000 7f096d400700 1 --2- 172.21.15.98:0/1419289469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c229b150f5ccca6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.468+0000 7f0967e00700 1 -- 172.21.15.98:0/1419289469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f095c004160 con 0x7f096807ddb0 2024-03-16T12:10:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f0967e00700 1 -- 172.21.15.98:0/1419289469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f095c004300 con 0x7f096807ddb0 2024-03-16T12:10:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f0967e00700 1 -- 172.21.15.98:0/1419289469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f095c011b50 con 0x7f096807ddb0 2024-03-16T12:10:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f096fe00700 1 -- 172.21.15.98:0/1419289469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 msgr2=0x7f096807c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f096fe00700 1 --2- 172.21.15.98:0/1419289469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f096fe00700 1 -- 172.21.15.98:0/1419289469 shutdown_connections 2024-03-16T12:10:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f096fe00700 1 --2- 172.21.15.98:0/1419289469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f096807c320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.469+0000 7f096fe00700 1 -- 172.21.15.98:0/1419289469 >> 172.21.15.98:0/1419289469 conn(0x7f09681032e0 msgr2=0x7f0968105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.470+0000 7f096fe00700 1 -- 172.21.15.98:0/1419289469 shutdown_connections 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.470+0000 7f096fe00700 1 -- 172.21.15.98:0/1419289469 wait complete. 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096fe00700 1 Processor -- start 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096fe00700 1 -- start start 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096fe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096fe00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0968109c60 con 0x7f096807ddb0 2024-03-16T12:10:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47112/0 (socket says 172.21.15.98:47112) 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.471+0000 7f096d400700 1 -- 172.21.15.98:0/4199853974 learned_addr learned my addr 172.21.15.98:0/4199853974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f096d400700 1 -- 172.21.15.98:0/4199853974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09681a2660 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f096d400700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f095c011b60 tx=0x7f095c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f095c003970 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f095c003b10 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f095c024610 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.472+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09681a28f0 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.473+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09681a55e0 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.474+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f095c01a430 con 0x7f096807ddb0 2024-03-16T12:10:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.474+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0968063900 con 0x7f096807ddb0 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.474+0000 7f0966000700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 0x7f095003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.474+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f095c011e00 con 0x7f096807ddb0 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.475+0000 7f096ca00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 0x7f095003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.475+0000 7f096ca00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 0x7f095003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0954006fd0 tx=0x7f0954006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.480+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f095c04ca20 con 0x7f096807ddb0 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.699+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f09680008d0 con 0x7f0950038f70 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.700+0000 7f0966000700 1 -- 172.21.15.98:0/4199853974 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f09680008d0 con 0x7f0950038f70 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.702+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 msgr2=0x7f095003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.702+0000 7f096fe00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 0x7f095003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0954006fd0 tx=0x7f0954006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.702+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 msgr2=0x7f09681a2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.702+0000 7f096fe00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f095c011b60 tx=0x7f095c011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.702+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 shutdown_connections 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.703+0000 7f096fe00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0950038f70 0x7f095003b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.703+0000 7f096fe00700 1 --2- 172.21.15.98:0/4199853974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096807ddb0 0x7f09681a2120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.703+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 >> 172.21.15.98:0/4199853974 conn(0x7f09681032e0 msgr2=0x7f0968198ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.703+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 shutdown_connections 2024-03-16T12:10:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:15.703+0000 7f096fe00700 1 -- 172.21.15.98:0/4199853974 wait complete. 2024-03-16T12:10:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:10:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T12:10:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-03-16T12:10:16.918 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:16 smithi098 ceph-mon[31633]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:17.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:17 smithi098 ceph-mon[31633]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:18.836 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:18 smithi098 ceph-mon[31633]: Deploying cephadm binary to smithi098 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.929+0000 7f027a600700 1 Processor -- start 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.929+0000 7f027a600700 1 -- start start 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.930+0000 7f027a600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.930+0000 7f027a600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f027407be10 con 0x7f02741064b0 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.930+0000 7f0273400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.930+0000 7f0273400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34250/0 (socket says 172.21.15.98:34250) 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.930+0000 7f0273400700 1 -- 172.21.15.98:0/4031866066 learned_addr learned my addr 172.21.15.98:0/4031866066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:24.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.931+0000 7f0273400700 1 -- 172.21.15.98:0/4031866066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0274106e10 con 0x7f02741064b0 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.931+0000 7f0273400700 1 --2- 172.21.15.98:0/4031866066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02640098a0 tx=0x7f0264009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae6aa49efda3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.931+0000 7f0272000700 1 -- 172.21.15.98:0/4031866066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0264004160 con 0x7f02741064b0 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.931+0000 7f0272000700 1 -- 172.21.15.98:0/4031866066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0264004300 con 0x7f02741064b0 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.931+0000 7f0272000700 1 -- 172.21.15.98:0/4031866066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0264011b50 con 0x7f02741064b0 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 -- 172.21.15.98:0/4031866066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 msgr2=0x7f02741068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 --2- 172.21.15.98:0/4031866066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02640098a0 tx=0x7f0264009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 -- 172.21.15.98:0/4031866066 shutdown_connections 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 --2- 172.21.15.98:0/4031866066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f02741068d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 -- 172.21.15.98:0/4031866066 >> 172.21.15.98:0/4031866066 conn(0x7f0274102090 msgr2=0x7f02741044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 -- 172.21.15.98:0/4031866066 shutdown_connections 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.932+0000 7f027a600700 1 -- 172.21.15.98:0/4031866066 wait complete. 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.933+0000 7f027a600700 1 Processor -- start 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f027a600700 1 -- start start 2024-03-16T12:10:24.137 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f027a600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f027a600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0274107e30 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f0273400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f0273400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34260/0 (socket says 172.21.15.98:34260) 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.934+0000 7f0273400700 1 -- 172.21.15.98:0/3345594514 learned_addr learned my addr 172.21.15.98:0/3345594514 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f0273400700 1 -- 172.21.15.98:0/3345594514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027419e2d0 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f0273400700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0264011b60 tx=0x7f0264011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0264003970 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0264003b10 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0264024610 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027419e560 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.935+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02741a1210 con 0x7f02741064b0 2024-03-16T12:10:24.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.937+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f026401a9f0 con 0x7f02741064b0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.937+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0274063900 con 0x7f02741064b0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.937+0000 7f026be00700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 0x7f025803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.937+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0264011e00 con 0x7f02741064b0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.938+0000 7f0272a00700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 0x7f025803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.938+0000 7f0272a00700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 0x7f025803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0260006fd0 tx=0x7f0260006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:16.942+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f026400b3e0 con 0x7f02741064b0 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:17.169+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7f027402df20 con 0x7f0258038f70 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.431+0000 7f026be00700 1 -- 172.21.15.98:0/3345594514 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f027402df20 con 0x7f0258038f70 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.435+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 msgr2=0x7f025803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.435+0000 7f027a600700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 0x7f025803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0260006fd0 tx=0x7f0260006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.435+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 msgr2=0x7f027419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.435+0000 7f027a600700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0264011b60 tx=0x7f0264011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 shutdown_connections 2024-03-16T12:10:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0258038f70 0x7f025803b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 --2- 172.21.15.98:0/3345594514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02741064b0 0x7f027419dd90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 >> 172.21.15.98:0/3345594514 conn(0x7f0274102090 msgr2=0x7f0274194a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 shutdown_connections 2024-03-16T12:10:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:23.436+0000 7f027a600700 1 -- 172.21.15.98:0/3345594514 wait complete. 2024-03-16T12:10:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-03-16T12:10:24.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:24 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:24.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:24 smithi098 ceph-mon[31633]: Added host smithi098 2024-03-16T12:10:24.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:24 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:26.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T12:10:26.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.723+0000 7fe37e400700 1 Processor -- start 2024-03-16T12:10:26.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.723+0000 7fe37e400700 1 -- start start 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe37e400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe37e400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3781085a0 con 0x7fe378107c40 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe377400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe377400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34266/0 (socket says 172.21.15.98:34266) 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe377400700 1 -- 172.21.15.98:0/3675506039 learned_addr learned my addr 172.21.15.98:0/3675506039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.724+0000 7fe377400700 1 -- 172.21.15.98:0/3675506039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe378108720 con 0x7fe378107c40 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.725+0000 7fe377400700 1 --2- 172.21.15.98:0/3675506039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3680098a0 tx=0x7fe368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ab4203344a59ae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.725+0000 7fe376000700 1 -- 172.21.15.98:0/3675506039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe368004160 con 0x7fe378107c40 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.725+0000 7fe376000700 1 -- 172.21.15.98:0/3675506039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe368004300 con 0x7fe378107c40 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.725+0000 7fe376000700 1 -- 172.21.15.98:0/3675506039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe368011b50 con 0x7fe378107c40 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 -- 172.21.15.98:0/3675506039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 msgr2=0x7fe378108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 --2- 172.21.15.98:0/3675506039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3680098a0 tx=0x7fe368009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 -- 172.21.15.98:0/3675506039 shutdown_connections 2024-03-16T12:10:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 --2- 172.21.15.98:0/3675506039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe378108060 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 -- 172.21.15.98:0/3675506039 >> 172.21.15.98:0/3675506039 conn(0x7fe3781032a0 msgr2=0x7fe3781056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 -- 172.21.15.98:0/3675506039 shutdown_connections 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.726+0000 7fe37e400700 1 -- 172.21.15.98:0/3675506039 wait complete. 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.727+0000 7fe37e400700 1 Processor -- start 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.727+0000 7fe37e400700 1 -- start start 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.727+0000 7fe37e400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.727+0000 7fe37e400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe378109be0 con 0x7fe378107c40 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.728+0000 7fe377400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.728+0000 7fe377400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34278/0 (socket says 172.21.15.98:34278) 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.728+0000 7fe377400700 1 -- 172.21.15.98:0/3830128975 learned_addr learned my addr 172.21.15.98:0/3830128975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.728+0000 7fe377400700 1 -- 172.21.15.98:0/3830128975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37807e090 con 0x7fe378107c40 2024-03-16T12:10:26.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.728+0000 7fe377400700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe368011b60 tx=0x7fe368011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.729+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe368003970 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.729+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe368003b10 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.729+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37807c280 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.729+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe368024610 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.729+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37807c7c0 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.730+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe36801a9f0 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.730+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe378063900 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.730+0000 7fe36fe00700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 0x7fe35c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.730+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe368011e00 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.731+0000 7fe376a00700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 0x7fe35c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.731+0000 7fe376a00700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 0x7fe35c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe364006fd0 tx=0x7fe364006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.736+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe36800b3e0 con 0x7fe378107c40 2024-03-16T12:10:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.961+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe37807d020 con 0x7fe35c038f70 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.967+0000 7fe36fe00700 1 -- 172.21.15.98:0/3830128975 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe37807d020 con 0x7fe35c038f70 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 msgr2=0x7fe35c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 0x7fe35c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe364006fd0 tx=0x7fe364006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 msgr2=0x7fe37807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe368011b60 tx=0x7fe368011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 shutdown_connections 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7fe35c038f70 0x7fe35c03b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 --2- 172.21.15.98:0/3830128975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe378107c40 0x7fe37807db50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.969+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 >> 172.21.15.98:0/3830128975 conn(0x7fe3781032a0 msgr2=0x7fe378103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.970+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 shutdown_connections 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:24.970+0000 7fe37e400700 1 -- 172.21.15.98:0/3830128975 wait complete. 2024-03-16T12:10:26.142 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-03-16T12:10:26.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:25 smithi098 ceph-mon[31633]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:26.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:25 smithi098 ceph-mon[31633]: Saving service mon spec with placement count:5 2024-03-16T12:10:26.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:25 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:28.156 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:28 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:28.156 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:28 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:28.157 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:28 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.572+0000 7f11ca800700 1 Processor -- start 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.572+0000 7f11ca800700 1 -- start start 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.573+0000 7f11ca800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.573+0000 7f11ca800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41085a0 con 0x7f11c4107c40 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.573+0000 7f11c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.573+0000 7f11c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42476/0 (socket says 172.21.15.98:42476) 2024-03-16T12:10:28.476 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.573+0000 7f11c3400700 1 -- 172.21.15.98:0/381112282 learned_addr learned my addr 172.21.15.98:0/381112282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.574+0000 7f11c3400700 1 -- 172.21.15.98:0/381112282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c4108720 con 0x7f11c4107c40 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.574+0000 7f11c3400700 1 --2- 172.21.15.98:0/381112282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11b40098a0 tx=0x7f11b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1d43f95841b78f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.574+0000 7f11c2000700 1 -- 172.21.15.98:0/381112282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11b4004160 con 0x7f11c4107c40 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.574+0000 7f11c2000700 1 -- 172.21.15.98:0/381112282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b4004300 con 0x7f11c4107c40 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.574+0000 7f11c2000700 1 -- 172.21.15.98:0/381112282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11b4011b20 con 0x7f11c4107c40 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.575+0000 7f11ca800700 1 -- 172.21.15.98:0/381112282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 msgr2=0x7f11c4108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.575+0000 7f11ca800700 1 --2- 172.21.15.98:0/381112282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11b40098a0 tx=0x7f11b4009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.575+0000 7f11ca800700 1 -- 172.21.15.98:0/381112282 shutdown_connections 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.575+0000 7f11ca800700 1 --2- 172.21.15.98:0/381112282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c4108060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.575+0000 7f11ca800700 1 -- 172.21.15.98:0/381112282 >> 172.21.15.98:0/381112282 conn(0x7f11c41032a0 msgr2=0x7f11c41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.576+0000 7f11ca800700 1 -- 172.21.15.98:0/381112282 shutdown_connections 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.576+0000 7f11ca800700 1 -- 172.21.15.98:0/381112282 wait complete. 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11ca800700 1 Processor -- start 2024-03-16T12:10:28.477 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11ca800700 1 -- start start 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11ca800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11ca800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410c060 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42486/0 (socket says 172.21.15.98:42486) 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.577+0000 7f11c3400700 1 -- 172.21.15.98:0/261743300 learned_addr learned my addr 172.21.15.98:0/261743300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.578+0000 7f11c3400700 1 -- 172.21.15.98:0/261743300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c407e090 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.578+0000 7f11c3400700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f11b4011b60 tx=0x7f11b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.579+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11b4003b60 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.579+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c407c280 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.579+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b401a910 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.579+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11b4024930 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.579+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c407c740 con 0x7f11c4107c40 2024-03-16T12:10:28.478 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.580+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f11b4022020 con 0x7f11c4107c40 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.580+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c4063900 con 0x7f11c4107c40 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.580+0000 7f11bbe00700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 0x7f11a803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.580+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f11b404d150 con 0x7f11c4107c40 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.581+0000 7f11c2a00700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 0x7f11a803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.581+0000 7f11c2a00700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 0x7f11a803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f11b0006fd0 tx=0x7f11b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.586+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11b4019c70 con 0x7f11c4107c40 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.822+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f11c407d050 con 0x7f11a8038f20 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.828+0000 7f11bbe00700 1 -- 172.21.15.98:0/261743300 <== mgr.14120 v2:172.21.15.98:6800/3396936170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f11c407d050 con 0x7f11a8038f20 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.830+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 msgr2=0x7f11a803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.830+0000 7f11ca800700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 0x7f11a803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f11b0006fd0 tx=0x7f11b0006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.830+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 msgr2=0x7f11c407db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.830+0000 7f11ca800700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f11b4011b60 tx=0x7f11b4011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:28.479 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 shutdown_connections 2024-03-16T12:10:28.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f11a8038f20 0x7f11a803b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:28.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 --2- 172.21.15.98:0/261743300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c4107c40 0x7f11c407db50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:28.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 >> 172.21.15.98:0/261743300 conn(0x7f11c41032a0 msgr2=0x7f11c4103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:28.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 shutdown_connections 2024-03-16T12:10:28.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:27.831+0000 7f11ca800700 1 -- 172.21.15.98:0/261743300 wait complete. 2024-03-16T12:10:29.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:29 smithi098 ceph-mon[31633]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:29.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:29 smithi098 ceph-mon[31633]: Saving service mgr spec with placement count:2 2024-03-16T12:10:29.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:29 smithi098 ceph-mon[31633]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:10:29.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/739351835' entity='client.admin' 2024-03-16T12:10:30.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.059+0000 7f90be600700 1 Processor -- start 2024-03-16T12:10:30.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.059+0000 7f90be600700 1 -- start start 2024-03-16T12:10:30.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.059+0000 7f90be600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:30.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.059+0000 7f90be600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90b807c890 con 0x7f90b807de60 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.060+0000 7f90b7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.060+0000 7f90b7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42496/0 (socket says 172.21.15.98:42496) 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.060+0000 7f90b7400700 1 -- 172.21.15.98:0/2505167481 learned_addr learned my addr 172.21.15.98:0/2505167481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.060+0000 7f90b7400700 1 -- 172.21.15.98:0/2505167481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b807ca10 con 0x7f90b807de60 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.060+0000 7f90b7400700 1 --2- 172.21.15.98:0/2505167481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90a80098a0 tx=0x7f90a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcb0a50a2f9e5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.061+0000 7f90b6000700 1 -- 172.21.15.98:0/2505167481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a8004160 con 0x7f90b807de60 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.061+0000 7f90b6000700 1 -- 172.21.15.98:0/2505167481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90a8004300 con 0x7f90b807de60 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.061+0000 7f90b6000700 1 -- 172.21.15.98:0/2505167481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a8011b20 con 0x7f90b807de60 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.061+0000 7f90be600700 1 -- 172.21.15.98:0/2505167481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 msgr2=0x7f90b807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.061+0000 7f90be600700 1 --2- 172.21.15.98:0/2505167481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90a80098a0 tx=0x7f90a8009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.062+0000 7f90be600700 1 -- 172.21.15.98:0/2505167481 shutdown_connections 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.062+0000 7f90be600700 1 --2- 172.21.15.98:0/2505167481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b807c2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.062+0000 7f90be600700 1 -- 172.21.15.98:0/2505167481 >> 172.21.15.98:0/2505167481 conn(0x7f90b8103160 msgr2=0x7f90b8105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.062+0000 7f90be600700 1 -- 172.21.15.98:0/2505167481 shutdown_connections 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.062+0000 7f90be600700 1 -- 172.21.15.98:0/2505167481 wait complete. 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90be600700 1 Processor -- start 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90be600700 1 -- start start 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90be600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90be600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90b810e220 con 0x7f90b807de60 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90b7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90b7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42512/0 (socket says 172.21.15.98:42512) 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.063+0000 7f90b7400700 1 -- 172.21.15.98:0/739351835 learned_addr learned my addr 172.21.15.98:0/739351835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90b7400700 1 -- 172.21.15.98:0/739351835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b819e1f0 con 0x7f90b807de60 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90b7400700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f90a8011b60 tx=0x7f90a8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a8003b60 con 0x7f90b807de60 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90a801a910 con 0x7f90b807de60 2024-03-16T12:10:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b819e480 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.064+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a8024930 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.065+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b819e8e0 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.066+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b8063900 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.066+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f90a8022020 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.066+0000 7f90afe00700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 0x7f909c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.066+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f90a804d080 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.067+0000 7f90b6a00700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 0x7f909c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.067+0000 7f90b6a00700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 0x7f909c03b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006fd0 tx=0x7f90a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.072+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90a800b3e0 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.292+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f90b81a16c0 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.297+0000 7f90afe00700 1 -- 172.21.15.98:0/739351835 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f90a8016260 con 0x7f90b807de60 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 msgr2=0x7f909c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:30.543 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 0x7f909c03b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006fd0 tx=0x7f90a4006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 msgr2=0x7f90b819dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f90a8011b60 tx=0x7f90a8011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 shutdown_connections 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f909c038f70 0x7f909c03b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 --2- 172.21.15.98:0/739351835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90b807de60 0x7f90b819dcb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 >> 172.21.15.98:0/739351835 conn(0x7f90b8103160 msgr2=0x7f90b8103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 shutdown_connections 2024-03-16T12:10:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:29.301+0000 7f90be600700 1 -- 172.21.15.98:0/739351835 wait complete. 2024-03-16T12:10:31.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:31 smithi098 ceph-mon[31633]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:10:31.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:31 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.106+0000 7f3e75400700 1 Processor -- start 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.107+0000 7f3e75400700 1 -- start start 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.107+0000 7f3e75400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.107+0000 7f3e75400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7007bdb0 con 0x7f3e70108720 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.108+0000 7f3e6e800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.108+0000 7f3e6e800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42522/0 (socket says 172.21.15.98:42522) 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.108+0000 7f3e6e800700 1 -- 172.21.15.98:0/293973770 learned_addr learned my addr 172.21.15.98:0/293973770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.108+0000 7f3e6e800700 1 -- 172.21.15.98:0/293973770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7010b050 con 0x7f3e70108720 2024-03-16T12:10:32.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.109+0000 7f3e6e800700 1 --2- 172.21.15.98:0/293973770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e600098a0 tx=0x7f3e60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcaba806209d5839 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.109+0000 7f3e6d400700 1 -- 172.21.15.98:0/293973770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e60004160 con 0x7f3e70108720 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.109+0000 7f3e6d400700 1 -- 172.21.15.98:0/293973770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e60004300 con 0x7f3e70108720 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.109+0000 7f3e6d400700 1 -- 172.21.15.98:0/293973770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e60011b20 con 0x7f3e70108720 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 -- 172.21.15.98:0/293973770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 msgr2=0x7f3e7010ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 --2- 172.21.15.98:0/293973770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e600098a0 tx=0x7f3e60009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 -- 172.21.15.98:0/293973770 shutdown_connections 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 --2- 172.21.15.98:0/293973770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7010ab10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 -- 172.21.15.98:0/293973770 >> 172.21.15.98:0/293973770 conn(0x7f3e70102090 msgr2=0x7f3e701044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 -- 172.21.15.98:0/293973770 shutdown_connections 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.110+0000 7f3e75400700 1 -- 172.21.15.98:0/293973770 wait complete. 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.112+0000 7f3e75400700 1 Processor -- start 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.112+0000 7f3e75400700 1 -- start start 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.112+0000 7f3e75400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.112+0000 7f3e75400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7010bff0 con 0x7f3e70108720 2024-03-16T12:10:32.051 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.113+0000 7f3e6e800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.113+0000 7f3e6e800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42528/0 (socket says 172.21.15.98:42528) 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.113+0000 7f3e6e800700 1 -- 172.21.15.98:0/3621791206 learned_addr learned my addr 172.21.15.98:0/3621791206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.113+0000 7f3e6e800700 1 -- 172.21.15.98:0/3621791206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7019e270 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.113+0000 7f3e6e800700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3e60011b60 tx=0x7f3e60011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.114+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e60003b60 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.114+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e6001a910 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.114+0000 7f3e75400700 1 -- 172.21.15.98:0/3621791206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7019e500 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.114+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e60024930 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.114+0000 7f3e75400700 1 -- 172.21.15.98:0/3621791206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7019e960 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.115+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3e60022020 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.115+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e70063900 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.116+0000 7f3e67400700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 0x7f3e5403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.116+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3e6004d140 con 0x7f3e70108720 2024-03-16T12:10:32.052 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.116+0000 7f3e6de00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 0x7f3e5403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.117+0000 7f3e6de00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 0x7f3e5403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c006fd0 tx=0x7f3e5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.121+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e60016260 con 0x7f3e70108720 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.346+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f3e70062ee0 con 0x7f3e70108720 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.349+0000 7f3e67400700 1 -- 172.21.15.98:0/3621791206 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f3e6004ca90 con 0x7f3e70108720 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 msgr2=0x7f3e5403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 0x7f3e5403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c006fd0 tx=0x7f3e5c006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 msgr2=0x7f3e7019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3e60011b60 tx=0x7f3e60011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 shutdown_connections 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f3e54038f20 0x7f3e5403b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 --2- 172.21.15.98:0/3621791206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e70108720 0x7f3e7019dd30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 >> 172.21.15.98:0/3621791206 conn(0x7f3e70102090 msgr2=0x7f3e70102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 shutdown_connections 2024-03-16T12:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:31.352+0000 7f3e64c00700 1 -- 172.21.15.98:0/3621791206 wait complete. 2024-03-16T12:10:32.054 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-03-16T12:10:32.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:32 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3621791206' entity='client.admin' 2024-03-16T12:10:33.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:33 smithi098 ceph-mon[31633]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:10:33.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:33 smithi098 ceph-mon[31633]: from='mgr.14120 172.21.15.98:0/2794766084' entity='mgr.x' 2024-03-16T12:10:33.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:33 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/195957770' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T12:10:34.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: ignoring --setuser ceph since I am not root 2024-03-16T12:10:34.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: ignoring --setgroup ceph since I am not root 2024-03-16T12:10:34.908 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:34.588+0000 7fcf5384e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T12:10:34.908 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:34.721+0000 7fcf5384e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T12:10:34.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:34 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/195957770' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T12:10:34.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:34 smithi098 ceph-mon[31633]: mgrmap e9: x(active, since 25s) 2024-03-16T12:10:34.915 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.706+0000 7f2b5c000700 1 Processor -- start 2024-03-16T12:10:34.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.706+0000 7f2b5c000700 1 -- start start 2024-03-16T12:10:34.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.706+0000 7f2b5c000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:34.916 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.706+0000 7f2b5c000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5410a7f0 con 0x7f2b54109e00 2024-03-16T12:10:34.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.707+0000 7f2b59600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:34.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.707+0000 7f2b59600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42534/0 (socket says 172.21.15.98:42534) 2024-03-16T12:10:34.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.707+0000 7f2b59600700 1 -- 172.21.15.98:0/3362923531 learned_addr learned my addr 172.21.15.98:0/3362923531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:34.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.707+0000 7f2b59600700 1 -- 172.21.15.98:0/3362923531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5410afb0 con 0x7f2b54109e00 2024-03-16T12:10:34.917 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.707+0000 7f2b59600700 1 --2- 172.21.15.98:0/3362923531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b480098a0 tx=0x7f2b48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda23aa5d4eab5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:34.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.708+0000 7f2b53e00700 1 -- 172.21.15.98:0/3362923531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b48004160 con 0x7f2b54109e00 2024-03-16T12:10:34.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.708+0000 7f2b53e00700 1 -- 172.21.15.98:0/3362923531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b48004300 con 0x7f2b54109e00 2024-03-16T12:10:34.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.708+0000 7f2b53e00700 1 -- 172.21.15.98:0/3362923531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b48011b20 con 0x7f2b54109e00 2024-03-16T12:10:34.918 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.708+0000 7f2b5c000700 1 -- 172.21.15.98:0/3362923531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 msgr2=0x7f2b5410a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:34.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.708+0000 7f2b5c000700 1 --2- 172.21.15.98:0/3362923531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b480098a0 tx=0x7f2b48009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:34.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.709+0000 7f2b5c000700 1 -- 172.21.15.98:0/3362923531 shutdown_connections 2024-03-16T12:10:34.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.709+0000 7f2b5c000700 1 --2- 172.21.15.98:0/3362923531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5410a220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:34.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.709+0000 7f2b5c000700 1 -- 172.21.15.98:0/3362923531 >> 172.21.15.98:0/3362923531 conn(0x7f2b5407d1d0 msgr2=0x7f2b5407d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:34.919 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.709+0000 7f2b5c000700 1 -- 172.21.15.98:0/3362923531 shutdown_connections 2024-03-16T12:10:34.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.709+0000 7f2b5c000700 1 -- 172.21.15.98:0/3362923531 wait complete. 2024-03-16T12:10:34.920 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b5c000700 1 Processor -- start 2024-03-16T12:10:34.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b5c000700 1 -- start start 2024-03-16T12:10:34.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b5c000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:34.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b5c000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5410e360 con 0x7f2b54109e00 2024-03-16T12:10:34.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b59600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:34.921 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b59600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42536/0 (socket says 172.21.15.98:42536) 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.710+0000 7f2b59600700 1 -- 172.21.15.98:0/195957770 learned_addr learned my addr 172.21.15.98:0/195957770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b59600700 1 -- 172.21.15.98:0/195957770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5419e260 con 0x7f2b54109e00 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b59600700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b480116a0 tx=0x7f2b48011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b48003a40 con 0x7f2b54109e00 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5419e4f0 con 0x7f2b54109e00 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b48003be0 con 0x7f2b54109e00 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b4801a720 con 0x7f2b54109e00 2024-03-16T12:10:34.924 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.711+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5419e950 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.712+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2b48024aa0 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.713+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b54063900 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.713+0000 7f2b52000700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 0x7f2b3c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.713+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b4804d4d0 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.713+0000 7f2b58c00700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 0x7f2b3c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.714+0000 7f2b58c00700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 0x7f2b3c03b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:32.719+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b48011e20 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.006+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f2b541a1410 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.753+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2b4804cde0 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.757+0000 7f2b52000700 1 -- 172.21.15.98:0/195957770 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f2b4802d770 con 0x7f2b54109e00 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 msgr2=0x7f2b3c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 0x7f2b3c03b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 msgr2=0x7f2b5419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b480116a0 tx=0x7f2b48011ba0 comp rx=0 tx=0).stop 2024-03-16T12:10:34.925 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 shutdown_connections 2024-03-16T12:10:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f2b3c038ed0 0x7f2b3c03b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 --2- 172.21.15.98:0/195957770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b54109e00 0x7f2b5419dd20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 >> 172.21.15.98:0/195957770 conn(0x7f2b5407d1d0 msgr2=0x7f2b541074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 shutdown_connections 2024-03-16T12:10:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:33.760+0000 7f2b5c000700 1 -- 172.21.15.98:0/195957770 wait complete. 2024-03-16T12:10:35.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:35.524+0000 7fcf5384e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T12:10:35.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:35.650+0000 7fcf5384e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T12:10:36.059 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:35 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2576516905' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T12:10:36.111 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:35.831+0000 7fcf5384e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T12:10:36.111 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:35.962+0000 7fcf5384e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T12:10:36.111 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:36.071+0000 7fcf5384e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.584+0000 7f0ada600700 1 Processor -- start 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.584+0000 7f0ada600700 1 -- start start 2024-03-16T12:10:36.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.585+0000 7f0ada600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.585+0000 7f0ada600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad407e140 con 0x7f0ad407dd20 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.585+0000 7f0ad3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.585+0000 7f0ad3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42556/0 (socket says 172.21.15.98:42556) 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.585+0000 7f0ad3400700 1 -- 172.21.15.98:0/2057977464 learned_addr learned my addr 172.21.15.98:0/2057977464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.586+0000 7f0ad3400700 1 -- 172.21.15.98:0/2057977464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad407c820 con 0x7f0ad407dd20 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.586+0000 7f0ad3400700 1 --2- 172.21.15.98:0/2057977464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40098a0 tx=0x7f0ac4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3df8ff6a5787a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.586+0000 7f0ad2000700 1 -- 172.21.15.98:0/2057977464 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac4004160 con 0x7f0ad407dd20 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.586+0000 7f0ad2000700 1 -- 172.21.15.98:0/2057977464 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac4004300 con 0x7f0ad407dd20 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.586+0000 7f0ad2000700 1 -- 172.21.15.98:0/2057977464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac4011b20 con 0x7f0ad407dd20 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 -- 172.21.15.98:0/2057977464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 msgr2=0x7f0ad407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 --2- 172.21.15.98:0/2057977464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40098a0 tx=0x7f0ac4009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 -- 172.21.15.98:0/2057977464 shutdown_connections 2024-03-16T12:10:36.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 --2- 172.21.15.98:0/2057977464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad407c2e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 -- 172.21.15.98:0/2057977464 >> 172.21.15.98:0/2057977464 conn(0x7f0ad41032a0 msgr2=0x7f0ad41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 -- 172.21.15.98:0/2057977464 shutdown_connections 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.587+0000 7f0ada600700 1 -- 172.21.15.98:0/2057977464 wait complete. 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.588+0000 7f0ada600700 1 Processor -- start 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.588+0000 7f0ada600700 1 -- start start 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.588+0000 7f0ada600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ada600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad410e0f0 con 0x7f0ad407dd20 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ad3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ad3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42572/0 (socket says 172.21.15.98:42572) 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ad3400700 1 -- 172.21.15.98:0/2576516905 learned_addr learned my addr 172.21.15.98:0/2576516905 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ad3400700 1 -- 172.21.15.98:0/2576516905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad419e270 con 0x7f0ad407dd20 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.589+0000 7f0ad3400700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40116a0 tx=0x7f0ac4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.590+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac4003a40 con 0x7f0ad407dd20 2024-03-16T12:10:36.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.590+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac4003be0 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.590+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac401a750 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.590+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad419e500 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.590+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad419e960 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.591+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad4063900 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.591+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f0ac4024aa0 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.592+0000 7f0acbe00700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 0x7f0ab803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.592+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0ac404d460 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.592+0000 7f0ad2a00700 1 -- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 msgr2=0x7f0ab803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.592+0000 7f0ad2a00700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 0x7f0ab803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.597+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ac4011e20 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.793+0000 7f0ad2a00700 1 -- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 msgr2=0x7f0ab803b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.793+0000 7f0ad2a00700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 0x7f0ab803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.874+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0ad41a17a0 con 0x7f0ad407dd20 2024-03-16T12:10:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.875+0000 7f0acbe00700 1 -- 172.21.15.98:0/2576516905 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0ac402d770 con 0x7f0ad407dd20 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 msgr2=0x7f0ab803b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 0x7f0ab803b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 msgr2=0x7f0ad419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40116a0 tx=0x7f0ac4011ba0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 shutdown_connections 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f0ab8038f80 0x7f0ab803b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 --2- 172.21.15.98:0/2576516905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ad407dd20 0x7f0ad419dd30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 >> 172.21.15.98:0/2576516905 conn(0x7f0ad41032a0 msgr2=0x7f0ad4103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.878+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 shutdown_connections 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:35.879+0000 7f0ada600700 1 -- 172.21.15.98:0/2576516905 wait complete. 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-03-16T12:10:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-03-16T12:10:36.655 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:36.408+0000 7fcf5384e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T12:10:36.655 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:36.552+0000 7fcf5384e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T12:10:36.656 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:36.667+0000 7fcf5384e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T12:10:36.921 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:36.797+0000 7fcf5384e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T12:10:37.249 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:37.023+0000 7fcf5384e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T12:10:38.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:38.279+0000 7fcf5384e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T12:10:38.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:38.447+0000 7fcf5384e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T12:10:39.235 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:38.875+0000 7fcf5384e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T12:10:39.235 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:38.992+0000 7fcf5384e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T12:10:39.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:39.246+0000 7fcf5384e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:10:39.499 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:39.489+0000 7fcf5384e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T12:10:40.165 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:39.860+0000 7fcf5384e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:10:40.165 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:40.176+0000 7fcf5384e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T12:10:40.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:40.477+0000 7fcf5384e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T12:10:41.749 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:41.492+0000 7fcf5384e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T12:10:42.311 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:42.198+0000 7fcf5384e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T12:10:42.618 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:42.334+0000 7fcf5384e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T12:10:42.998 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:10:42.629+0000 7fcf5384e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: Active manager daemon x restarted 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: Activating manager daemon x 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: mgrmap e10: x(active, starting, since 0.0056849s) 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T12:10:42.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: Manager daemon x is now available 2024-03-16T12:10:43.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:43.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:10:43.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T12:10:43.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T12:10:43.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:43 smithi098 ceph-mon[31633]: mgrmap e11: x(active, since 1.01214s) 2024-03-16T12:10:44.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-03-16T12:10:44.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T12:10:44.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T12:10:44.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.087+0000 7f9341000700 1 Processor -- start 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.087+0000 7f9341000700 1 -- start start 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.088+0000 7f9341000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.088+0000 7f9341000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f933c108560 con 0x7f933c107b70 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.088+0000 7f9339e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.088+0000 7f9339e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34610/0 (socket says 172.21.15.98:34610) 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.088+0000 7f9339e00700 1 -- 172.21.15.98:0/963829728 learned_addr learned my addr 172.21.15.98:0/963829728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.089+0000 7f9339e00700 1 -- 172.21.15.98:0/963829728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c108d20 con 0x7f933c107b70 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.089+0000 7f9339e00700 1 --2- 172.21.15.98:0/963829728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f932c0098a0 tx=0x7f932c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974ae6e60fd0d737 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.089+0000 7f9338a00700 1 -- 172.21.15.98:0/963829728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c004160 con 0x7f933c107b70 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.089+0000 7f9338a00700 1 -- 172.21.15.98:0/963829728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f932c004300 con 0x7f933c107b70 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.089+0000 7f9338a00700 1 -- 172.21.15.98:0/963829728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c011b20 con 0x7f933c107b70 2024-03-16T12:10:44.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 -- 172.21.15.98:0/963829728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 msgr2=0x7f933c107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 --2- 172.21.15.98:0/963829728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f932c0098a0 tx=0x7f932c009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 -- 172.21.15.98:0/963829728 shutdown_connections 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 --2- 172.21.15.98:0/963829728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c107f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 -- 172.21.15.98:0/963829728 >> 172.21.15.98:0/963829728 conn(0x7f933c1030d0 msgr2=0x7f933c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 -- 172.21.15.98:0/963829728 shutdown_connections 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.090+0000 7f9341000700 1 -- 172.21.15.98:0/963829728 wait complete. 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.091+0000 7f9341000700 1 Processor -- start 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9341000700 1 -- start start 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9341000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9341000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f933c10c0d0 con 0x7f933c107b70 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9339e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9339e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34622/0 (socket says 172.21.15.98:34622) 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.092+0000 7f9339e00700 1 -- 172.21.15.98:0/738137490 learned_addr learned my addr 172.21.15.98:0/738137490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:44.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9339e00700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c19c030 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9339e00700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f932c000c00 tx=0x7f932c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c003b60 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f932c01a910 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c024930 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933c19c2c0 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.093+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933c19c720 con 0x7f933c107b70 2024-03-16T12:10:44.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.094+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f932c022020 con 0x7f933c107b70 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.095+0000 7f9332a00700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.095+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f932003bba0 con 0x7f9320038fd0 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.095+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f932c04d160 con 0x7f933c107b70 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.095+0000 7f9339400700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.095+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.296+0000 7f9339400700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.296+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.696+0000 7f9339400700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:44.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:37.696+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:38.497+0000 7f9339400700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:38.497+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:40.099+0000 7f9339400700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3396936170 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:40.099+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:42.636+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f932c024ad0 con 0x7f933c107b70 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:42.636+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 msgr2=0x7f932003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:42.636+0000 7f9332a00700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.640+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f932c04ead0 con 0x7f933c107b70 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.640+0000 7f9332a00700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 0x7f932003ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.640+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f932c00af30 con 0x7f932003ca80 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.641+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 0x7f932003ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.643+0000 7f9339400700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 0x7f932003ee70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9328003a10 tx=0x7f9328009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.644+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f932c00af30 con 0x7f932003ca80 2024-03-16T12:10:44.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.651+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f933c063900 con 0x7f932003ca80 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9332a00700 1 -- 172.21.15.98:0/738137490 <== mgr.14152 v2:172.21.15.98:6800/3089653509 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f933c063900 con 0x7f932003ca80 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 msgr2=0x7f932003ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 0x7f932003ee70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9328003a10 tx=0x7f9328009270 comp rx=0 tx=0).stop 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 msgr2=0x7f933c19baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f932c000c00 tx=0x7f932c011c10 comp rx=0 tx=0).stop 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 shutdown_connections 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f932003ca80 0x7f932003ee70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f933c107b70 0x7f933c19baf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 --2- 172.21.15.98:0/738137490 >> [v2:172.21.15.98:6800/3396936170,v1:172.21.15.98:6801/3396936170] conn(0x7f9320038fd0 0x7f932003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 >> 172.21.15.98:0/738137490 conn(0x7f933c1030d0 msgr2=0x7f933c103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 shutdown_connections 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:43.652+0000 7f9341000700 1 -- 172.21.15.98:0/738137490 wait complete. 2024-03-16T12:10:44.302 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-03-16T12:10:44.303 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:42] ENGINE Bus STARTING 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:43] ENGINE Serving on http://172.21.15.98:8765 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:43] ENGINE Serving on https://172.21.15.98:7150 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: [16/Mar/2024:12:10:43] ENGINE Bus STARTED 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T12:10:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:44 smithi098 ceph-mon[31633]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T12:10:45.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:45 smithi098 ceph-mon[31633]: mgrmap e12: x(active, since 2s) 2024-03-16T12:10:45.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:10:46.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T12:10:46.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.402+0000 7f51f3000700 1 Processor -- start 2024-03-16T12:10:46.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.402+0000 7f51f3000700 1 -- start start 2024-03-16T12:10:46.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.402+0000 7f51f3000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:46.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.402+0000 7f51f3000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1085a0 con 0x7f51ec107c40 2024-03-16T12:10:46.444 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.403+0000 7f51ebe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.403+0000 7f51ebe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34696/0 (socket says 172.21.15.98:34696) 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.403+0000 7f51ebe00700 1 -- 172.21.15.98:0/3334939941 learned_addr learned my addr 172.21.15.98:0/3334939941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.403+0000 7f51ebe00700 1 -- 172.21.15.98:0/3334939941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec108720 con 0x7f51ec107c40 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.404+0000 7f51ebe00700 1 --2- 172.21.15.98:0/3334939941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098a0 tx=0x7f51dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94689e746f55cef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.404+0000 7f51eaa00700 1 -- 172.21.15.98:0/3334939941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc004160 con 0x7f51ec107c40 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.404+0000 7f51eaa00700 1 -- 172.21.15.98:0/3334939941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51dc004300 con 0x7f51ec107c40 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.404+0000 7f51eaa00700 1 -- 172.21.15.98:0/3334939941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc011b20 con 0x7f51ec107c40 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 -- 172.21.15.98:0/3334939941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 msgr2=0x7f51ec108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:46.445 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 --2- 172.21.15.98:0/3334939941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098a0 tx=0x7f51dc009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 -- 172.21.15.98:0/3334939941 shutdown_connections 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 --2- 172.21.15.98:0/3334939941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec108060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 -- 172.21.15.98:0/3334939941 >> 172.21.15.98:0/3334939941 conn(0x7f51ec1032a0 msgr2=0x7f51ec1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 -- 172.21.15.98:0/3334939941 shutdown_connections 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.405+0000 7f51f3000700 1 -- 172.21.15.98:0/3334939941 wait complete. 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.406+0000 7f51f3000700 1 Processor -- start 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51f3000700 1 -- start start 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51f3000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51f3000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec10c060 con 0x7f51ec107c40 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51ebe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51ebe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34700/0 (socket says 172.21.15.98:34700) 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.407+0000 7f51ebe00700 1 -- 172.21.15.98:0/2906778326 learned_addr learned my addr 172.21.15.98:0/2906778326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:46.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.408+0000 7f51ebe00700 1 -- 172.21.15.98:0/2906778326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec19e200 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.408+0000 7f51ebe00700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51dc011b60 tx=0x7f51dc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.408+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc003b60 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.408+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51dc01a910 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.408+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec19e490 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.409+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc024930 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.409+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec19e8f0 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.410+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec063900 con 0x7f51ec107c40 2024-03-16T12:10:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.410+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f51dc022020 con 0x7f51ec107c40 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.411+0000 7f51e8c00700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 0x7f51d003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.411+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f51dc04d2e0 con 0x7f51ec107c40 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.411+0000 7f51eb400700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 0x7f51d003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.412+0000 7f51eb400700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 0x7f51d003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.417+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51dc00b5a0 con 0x7f51ec107c40 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.644+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f51ec1043a0 con 0x7f51d0038f70 2024-03-16T12:10:46.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.790+0000 7f51e8c00700 1 -- 172.21.15.98:0/2906778326 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f51ec1043a0 con 0x7f51d0038f70 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 msgr2=0x7f51d003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 0x7f51d003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 msgr2=0x7f51ec19dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51dc011b60 tx=0x7f51dc011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 shutdown_connections 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f51d0038f70 0x7f51d003b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 --2- 172.21.15.98:0/2906778326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ec107c40 0x7f51ec19dcc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.793+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 >> 172.21.15.98:0/2906778326 conn(0x7f51ec1032a0 msgr2=0x7f51ec103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.794+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 shutdown_connections 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:45.794+0000 7f51f3000700 1 -- 172.21.15.98:0/2906778326 wait complete. 2024-03-16T12:10:46.452 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-03-16T12:10:47.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:46 smithi098 ceph-mon[31633]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:47.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:46 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:10:47.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:46 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:10:48.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$fk2BBthLdBsVU.QC.CRRyebseRVH6bCZdbzDXeJCTi/gBFrcTnMIO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710591047, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T12:10:48.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.106+0000 7fb0ea200700 1 Processor -- start 2024-03-16T12:10:48.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.106+0000 7fb0ea200700 1 -- start start 2024-03-16T12:10:48.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.106+0000 7fb0ea200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.106+0000 7fb0ea200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e407be10 con 0x7fb0e41064a0 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.107+0000 7fb0e3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.107+0000 7fb0e3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47002/0 (socket says 172.21.15.98:47002) 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.107+0000 7fb0e3400700 1 -- 172.21.15.98:0/367630255 learned_addr learned my addr 172.21.15.98:0/367630255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.108+0000 7fb0e3400700 1 -- 172.21.15.98:0/367630255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e4106e00 con 0x7fb0e41064a0 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.108+0000 7fb0e3400700 1 --2- 172.21.15.98:0/367630255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40098a0 tx=0x7fb0d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3eb38c160240b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.108+0000 7fb0e2000700 1 -- 172.21.15.98:0/367630255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d4004160 con 0x7fb0e41064a0 2024-03-16T12:10:48.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.108+0000 7fb0e2000700 1 -- 172.21.15.98:0/367630255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d4004300 con 0x7fb0e41064a0 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.108+0000 7fb0e2000700 1 -- 172.21.15.98:0/367630255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d4011b20 con 0x7fb0e41064a0 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.109+0000 7fb0ea200700 1 -- 172.21.15.98:0/367630255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 msgr2=0x7fb0e41068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.109+0000 7fb0ea200700 1 --2- 172.21.15.98:0/367630255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40098a0 tx=0x7fb0d4009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.109+0000 7fb0ea200700 1 -- 172.21.15.98:0/367630255 shutdown_connections 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.109+0000 7fb0ea200700 1 --2- 172.21.15.98:0/367630255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e41068c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.109+0000 7fb0ea200700 1 -- 172.21.15.98:0/367630255 >> 172.21.15.98:0/367630255 conn(0x7fb0e41020a0 msgr2=0x7fb0e41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:48.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.110+0000 7fb0ea200700 1 -- 172.21.15.98:0/367630255 shutdown_connections 2024-03-16T12:10:48.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.110+0000 7fb0ea200700 1 -- 172.21.15.98:0/367630255 wait complete. 2024-03-16T12:10:48.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.111+0000 7fb0ea200700 1 Processor -- start 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.111+0000 7fb0ea200700 1 -- start start 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.111+0000 7fb0ea200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.111+0000 7fb0ea200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e4107ed0 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.112+0000 7fb0e3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.112+0000 7fb0e3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47006/0 (socket says 172.21.15.98:47006) 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.112+0000 7fb0e3400700 1 -- 172.21.15.98:0/1121656272 learned_addr learned my addr 172.21.15.98:0/1121656272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.112+0000 7fb0e3400700 1 -- 172.21.15.98:0/1121656272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e419e200 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.112+0000 7fb0e3400700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4011b60 tx=0x7fb0d4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.113+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d4003b60 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.113+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d401a910 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.113+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e419e490 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.113+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d4024930 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.113+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e419e8f0 con 0x7fb0e41064a0 2024-03-16T12:10:48.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.115+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e4063900 con 0x7fb0e41064a0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.115+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fb0d4022020 con 0x7fb0e41064a0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.116+0000 7fb0dbe00700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 0x7fb0c803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.116+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb0d404d6e0 con 0x7fb0e41064a0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.116+0000 7fb0e2a00700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 0x7fb0c803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.117+0000 7fb0e2a00700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 0x7fb0c803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006fd0 tx=0x7fb0d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.121+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0d400bcb0 con 0x7fb0e41064a0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.368+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fb0e4197780 con 0x7fb0c8038fd0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.601+0000 7fb0dbe00700 1 -- 172.21.15.98:0/1121656272 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fb0e4197780 con 0x7fb0c8038fd0 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 msgr2=0x7fb0c803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 0x7fb0c803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006fd0 tx=0x7fb0d0006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 msgr2=0x7fb0e419dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4011b60 tx=0x7fb0d4011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 shutdown_connections 2024-03-16T12:10:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0c8038fd0 0x7fb0c803b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 --2- 172.21.15.98:0/1121656272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e41064a0 0x7fb0e419dcc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 >> 172.21.15.98:0/1121656272 conn(0x7fb0e41020a0 msgr2=0x7fb0e41948c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 shutdown_connections 2024-03-16T12:10:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:47.604+0000 7fb0ea200700 1 -- 172.21.15.98:0/1121656272 wait complete. 2024-03-16T12:10:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-03-16T12:10:48.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:10:48.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:48 smithi098 ceph-mon[31633]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:10:48.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.110+0000 7f95e0a00700 1 Processor -- start 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.110+0000 7f95e0a00700 1 -- start start 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95e0a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95e0a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc1085a0 con 0x7f95dc107c40 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95d9e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95d9e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47014/0 (socket says 172.21.15.98:47014) 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95d9e00700 1 -- 172.21.15.98:0/632693115 learned_addr learned my addr 172.21.15.98:0/632693115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:50.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.111+0000 7f95d9e00700 1 -- 172.21.15.98:0/632693115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc108720 con 0x7f95dc107c40 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.112+0000 7f95d9e00700 1 --2- 172.21.15.98:0/632693115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0098a0 tx=0x7f95cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f68b2168766560e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.112+0000 7f95d8a00700 1 -- 172.21.15.98:0/632693115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95cc004160 con 0x7f95dc107c40 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.112+0000 7f95d8a00700 1 -- 172.21.15.98:0/632693115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95cc004300 con 0x7f95dc107c40 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.112+0000 7f95d8a00700 1 -- 172.21.15.98:0/632693115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95cc011b20 con 0x7f95dc107c40 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 -- 172.21.15.98:0/632693115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 msgr2=0x7f95dc108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 --2- 172.21.15.98:0/632693115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0098a0 tx=0x7f95cc009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 -- 172.21.15.98:0/632693115 shutdown_connections 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 --2- 172.21.15.98:0/632693115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc108060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 -- 172.21.15.98:0/632693115 >> 172.21.15.98:0/632693115 conn(0x7f95dc1032a0 msgr2=0x7f95dc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 -- 172.21.15.98:0/632693115 shutdown_connections 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.113+0000 7f95e0a00700 1 -- 172.21.15.98:0/632693115 wait complete. 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.114+0000 7f95e0a00700 1 Processor -- start 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95e0a00700 1 -- start start 2024-03-16T12:10:50.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95e0a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95e0a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc10c060 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95d9e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95d9e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47026/0 (socket says 172.21.15.98:47026) 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95d9e00700 1 -- 172.21.15.98:0/498490229 learned_addr learned my addr 172.21.15.98:0/498490229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.115+0000 7f95d9e00700 1 -- 172.21.15.98:0/498490229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc07e090 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.116+0000 7f95d9e00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95cc011b60 tx=0x7f95cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.116+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95cc003b60 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.116+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95dc07c280 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.116+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95cc01a910 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.116+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95dc07c740 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.117+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95cc024930 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.118+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f95cc022020 con 0x7f95dc107c40 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.118+0000 7f95d2a00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 0x7f95c003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:50.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.118+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f95cc04e1d0 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.118+0000 7f95d9400700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 0x7f95c003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.118+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc005360 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.119+0000 7f95d9400700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 0x7f95c003b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95c8006fd0 tx=0x7f95c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.125+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f95cc02d910 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.268+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f95cc02d3f0 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.350+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f95bc005650 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.351+0000 7f95d2a00700 1 -- 172.21.15.98:0/498490229 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f95cc050f70 con 0x7f95dc107c40 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 msgr2=0x7f95c003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 0x7f95c003b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95c8006fd0 tx=0x7f95c8006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 msgr2=0x7f95dc07db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95cc011b60 tx=0x7f95cc011c40 comp rx=0 tx=0).stop 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 shutdown_connections 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95c0039020 0x7f95c003b4e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:50.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 --2- 172.21.15.98:0/498490229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95dc107c40 0x7f95dc07db50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:50.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 >> 172.21.15.98:0/498490229 conn(0x7f95dc1032a0 msgr2=0x7f95dc103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:50.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 shutdown_connections 2024-03-16T12:10:50.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:49.354+0000 7f95e0a00700 1 -- 172.21.15.98:0/498490229 wait complete. 2024-03-16T12:10:50.254 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T12:10:50.254 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-03-16T12:10:50.278 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T12:10:50.278 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-03-16T12:10:50.278 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-03-16T12:10:50.279 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout: Password: msy99ijiu3 2024-03-16T12:10:50.286 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:50.287 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config directory 2024-03-16T12:10:50.290 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-03-16T12:10:50.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:50 smithi098 ceph-mon[31633]: mgrmap e13: x(active, since 6s) 2024-03-16T12:10:50.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:50 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/498490229' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T12:10:52.074 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.181+0000 7fb3a1000700 1 Processor -- start 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.181+0000 7fb3a1000700 1 -- start start 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.181+0000 7fb3a1000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.181+0000 7fb3a1000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c10a760 con 0x7fb39c109d70 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.182+0000 7fb399e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.182+0000 7fb399e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47032/0 (socket says 172.21.15.98:47032) 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.182+0000 7fb399e00700 1 -- 172.21.15.98:0/134061880 learned_addr learned my addr 172.21.15.98:0/134061880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:52.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.183+0000 7fb399e00700 1 -- 172.21.15.98:0/134061880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c10af20 con 0x7fb39c109d70 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.183+0000 7fb399e00700 1 --2- 172.21.15.98:0/134061880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0098a0 tx=0x7fb38c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b7e682e21f0964 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.183+0000 7fb399400700 1 -- 172.21.15.98:0/134061880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb38c004160 con 0x7fb39c109d70 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.183+0000 7fb399400700 1 -- 172.21.15.98:0/134061880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c004300 con 0x7fb39c109d70 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.183+0000 7fb399400700 1 -- 172.21.15.98:0/134061880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb38c011b20 con 0x7fb39c109d70 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 -- 172.21.15.98:0/134061880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 msgr2=0x7fb39c10a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 --2- 172.21.15.98:0/134061880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0098a0 tx=0x7fb38c009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 -- 172.21.15.98:0/134061880 shutdown_connections 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 --2- 172.21.15.98:0/134061880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c10a190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 -- 172.21.15.98:0/134061880 >> 172.21.15.98:0/134061880 conn(0x7fb39c07d170 msgr2=0x7fb39c07d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 -- 172.21.15.98:0/134061880 shutdown_connections 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.184+0000 7fb3a1000700 1 -- 172.21.15.98:0/134061880 wait complete. 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.185+0000 7fb3a1000700 1 Processor -- start 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.185+0000 7fb3a1000700 1 -- start start 2024-03-16T12:10:52.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb3a1000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb3a1000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c10e2d0 con 0x7fb39c109d70 2024-03-16T12:10:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb399e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb399e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47034/0 (socket says 172.21.15.98:47034) 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb399e00700 1 -- 172.21.15.98:0/1069103149 learned_addr learned my addr 172.21.15.98:0/1069103149 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.186+0000 7fb399e00700 1 -- 172.21.15.98:0/1069103149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c19e250 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb399e00700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb38c003b60 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c01a910 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c19e4e0 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb38c024930 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.187+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c19e940 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.188+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb38c022020 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.188+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39c063900 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.189+0000 7fb392a00700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 0x7fb38003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.189+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb38c04d220 con 0x7fb39c109d70 2024-03-16T12:10:52.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.189+0000 7fb393e00700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 0x7fb38003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.190+0000 7fb393e00700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 0x7fb38003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb384006fd0 tx=0x7fb384006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.194+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb38c02d3f0 con 0x7fb39c109d70 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.414+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb39c1a13a0 con 0x7fb39c109d70 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.415+0000 7fb392a00700 1 -- 172.21.15.98:0/1069103149 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb38c01a430 con 0x7fb39c109d70 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 msgr2=0x7fb38003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 0x7fb38003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb384006fd0 tx=0x7fb384006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 msgr2=0x7fb39c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c011c10 comp rx=0 tx=0).stop 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 shutdown_connections 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb380039070 0x7fb38003b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 --2- 172.21.15.98:0/1069103149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb39c109d70 0x7fb39c19dd10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.417+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 >> 172.21.15.98:0/1069103149 conn(0x7fb39c07d170 msgr2=0x7fb39c1074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.418+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 shutdown_connections 2024-03-16T12:10:52.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:51.418+0000 7fb3a1000700 1 -- 172.21.15.98:0/1069103149 wait complete. 2024-03-16T12:10:53.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.673+0000 7f2de9c00700 1 Processor -- start 2024-03-16T12:10:53.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.673+0000 7f2de9c00700 1 -- start start 2024-03-16T12:10:53.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.673+0000 7f2de9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:53.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.673+0000 7f2de9c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4108610 con 0x7f2de4107c20 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.674+0000 7f2de2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.674+0000 7f2de2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47048/0 (socket says 172.21.15.98:47048) 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.674+0000 7f2de2a00700 1 -- 172.21.15.98:0/3793812597 learned_addr learned my addr 172.21.15.98:0/3793812597 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.674+0000 7f2de2a00700 1 -- 172.21.15.98:0/3793812597 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de4108d80 con 0x7f2de4107c20 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.675+0000 7f2de2a00700 1 --2- 172.21.15.98:0/3793812597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40098a0 tx=0x7f2dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d690cebe1fc38b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.675+0000 7f2de1600700 1 -- 172.21.15.98:0/3793812597 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dd4004160 con 0x7f2de4107c20 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.675+0000 7f2de1600700 1 -- 172.21.15.98:0/3793812597 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd4004300 con 0x7f2de4107c20 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.675+0000 7f2de1600700 1 -- 172.21.15.98:0/3793812597 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dd4011b20 con 0x7f2de4107c20 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.675+0000 7f2de9c00700 1 -- 172.21.15.98:0/3793812597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 msgr2=0x7f2de4108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:53.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 --2- 172.21.15.98:0/3793812597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40098a0 tx=0x7f2dd4009bb0 comp rx=0 tx=0).stop 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 -- 172.21.15.98:0/3793812597 shutdown_connections 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 --2- 172.21.15.98:0/3793812597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de4108040 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 -- 172.21.15.98:0/3793812597 >> 172.21.15.98:0/3793812597 conn(0x7f2de41031c0 msgr2=0x7f2de4105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 -- 172.21.15.98:0/3793812597 shutdown_connections 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.676+0000 7f2de9c00700 1 -- 172.21.15.98:0/3793812597 wait complete. 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.677+0000 7f2de9c00700 1 Processor -- start 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.677+0000 7f2de9c00700 1 -- start start 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.677+0000 7f2de9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de9c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2de410c130 con 0x7f2de4107c20 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47058/0 (socket says 172.21.15.98:47058) 2024-03-16T12:10:53.747 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de2a00700 1 -- 172.21.15.98:0/2793960788 learned_addr learned my addr 172.21.15.98:0/2793960788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de2a00700 1 -- 172.21.15.98:0/2793960788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de419c050 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.678+0000 7f2de2a00700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4000c00 tx=0x7f2dd4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.679+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dd4003b60 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.679+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd401a910 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.679+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de419c2e0 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.679+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dd4024930 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.679+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de419c740 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.680+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de4063900 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.680+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f2dd4022020 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.680+0000 7f2ddb400700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 0x7f2dc803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.681+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2dd404d2e0 con 0x7f2de4107c20 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.681+0000 7f2de2000700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 0x7f2dc803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:53.748 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.681+0000 7f2de2000700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 0x7f2dc803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0006fd0 tx=0x7f2dd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:52.686+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2dd402d3f0 con 0x7f2de4107c20 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.020+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f2de419f110 con 0x7f2de4107c20 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.025+0000 7f2ddb400700 1 -- 172.21.15.98:0/2793960788 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f2dd401a430 con 0x7f2de4107c20 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 msgr2=0x7f2dc803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 0x7f2dc803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0006fd0 tx=0x7f2dd0006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 msgr2=0x7f2de419bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4000c00 tx=0x7f2dd4011c10 comp rx=0 tx=0).stop 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 shutdown_connections 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2dc8039070 0x7f2dc803b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 --2- 172.21.15.98:0/2793960788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2de4107c20 0x7f2de419bb10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 >> 172.21.15.98:0/2793960788 conn(0x7f2de41031c0 msgr2=0x7f2de4103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 shutdown_connections 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-03-16T12:10:53.028+0000 7f2de9c00700 1 -- 172.21.15.98:0/2793960788 wait complete. 2024-03-16T12:10:53.749 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-03-16T12:10:53.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.751 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T12:10:53.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:10:53.751 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-03-16T12:10:53.833 INFO:tasks.cephadm:Fetching config... 2024-03-16T12:10:53.833 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:10:53.833 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T12:10:53.883 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T12:10:53.883 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:10:53.883 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T12:10:53.967 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T12:10:53.968 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:10:53.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T12:10:54.097 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T12:10:54.097 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:10:54.097 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T12:10:54.192 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T12:10:54.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T12:10:54.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:54 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2793960788' entity='client.admin' 2024-03-16T12:10:54.328 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:10:54.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T12:10:54.441 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:10:54.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T12:10:54.508 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxWZUZunlGkeqY8EREGo74mmK+FaeFxD1vJkUbQEGe5btdHV+aSAPQ/06Z6URrhiq9TkrVEsPo8ejQyTZdAm0iKGTIyjBUS5LhOFTjYCSUObo3SFNaAfqfG6hGQvB2a/ysGsGxfQucd1S1E6Rm43GIVJLlN5r5PbKcUX4gwlL/chnd4uU5j2/Uvyz1nF623R6eu1xJTLYP1hp5uuex7R2eEld4zFjpi32pbxHb3CX6jaUBIlq9U0rVZWRu/bPuWQHwqpMYOvBb+MbU8hd9SB1oAz0xmY3k43LBY6OCqZyxUUOV6BWRDkY+yct5bf2MWF9gVmXJ7DGOOwfIg6ucx9TcUWD7wVSEbjsM3dcTx4qI7DqUhn90iWv0y3rtO8lOT7C5sM/3kNNZ81au9Def6GZ/YwE377Lu09NfcwMvhYeTeCI+wocxl4h4odGtLU5tBVUmVGyzOIF8DhJ7RgJU2ZvDIW2id52jeqEPI3w416N+S2fN6ljt2EWjm6xsO0ksquU= ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:10:54.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T12:10:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 -- 172.21.15.98:0/2472559651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 msgr2=0x7f4664102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 --2- 172.21.15.98:0/2472559651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664102120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4658009910 tx=0x7f4658009c20 comp rx=0 tx=0).stop 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 -- 172.21.15.98:0/2472559651 shutdown_connections 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 --2- 172.21.15.98:0/2472559651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664102120 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 -- 172.21.15.98:0/2472559651 >> 172.21.15.98:0/2472559651 conn(0x7f46640fd2a0 msgr2=0x7f46640ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 -- 172.21.15.98:0/2472559651 shutdown_connections 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.032+0000 7f466ba00700 1 -- 172.21.15.98:0/2472559651 wait complete. 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.033+0000 7f466ba00700 1 Processor -- start 2024-03-16T12:10:57.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.033+0000 7f466ba00700 1 -- start start 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f466ba00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f466ba00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4664107d90 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f4669000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f4669000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38840/0 (socket says 172.21.15.98:38840) 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f4669000700 1 -- 172.21.15.98:0/3164269795 learned_addr learned my addr 172.21.15.98:0/3164269795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.034+0000 7f4669000700 1 -- 172.21.15.98:0/3164269795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4664197940 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f4669000700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4658005850 tx=0x7f4658005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4658026070 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46580044a0 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4658003ab0 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4664197bd0 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.035+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466419a8c0 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.036+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4658029070 con 0x7f4664101d00 2024-03-16T12:10:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.036+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4664108510 con 0x7f4664101d00 2024-03-16T12:10:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.037+0000 7f4661600700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 0x7f464c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:10:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.037+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f465801e070 con 0x7f4664101d00 2024-03-16T12:10:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.037+0000 7f4663e00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 0x7f464c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:10:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.037+0000 7f4663e00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 0x7f464c03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4650006fd0 tx=0x7f4650006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:10:57.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.043+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4658013070 con 0x7f4664101d00 2024-03-16T12:10:57.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.266+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f466402dc80 con 0x7f4664101d00 2024-03-16T12:10:57.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.272+0000 7f4661600700 1 -- 172.21.15.98:0/3164269795 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4658030020 con 0x7f4664101d00 2024-03-16T12:10:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 msgr2=0x7f464c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 0x7f464c03b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4650006fd0 tx=0x7f4650006e40 comp rx=0 tx=0).stop 2024-03-16T12:10:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 msgr2=0x7f4664197400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:10:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4658005850 tx=0x7f4658005030 comp rx=0 tx=0).stop 2024-03-16T12:10:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 shutdown_connections 2024-03-16T12:10:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f464c039020 0x7f464c03b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 --2- 172.21.15.98:0/3164269795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4664101d00 0x7f4664197400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:10:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.275+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 >> 172.21.15.98:0/3164269795 conn(0x7f46640fd2a0 msgr2=0x7f46640fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:10:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.276+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 shutdown_connections 2024-03-16T12:10:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:10:57.276+0000 7f466ba00700 1 -- 172.21.15.98:0/3164269795 wait complete. 2024-03-16T12:10:58.061 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T12:10:58.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T12:10:58.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:10:58 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3164269795' entity='client.admin' 2024-03-16T12:10:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 -- 172.21.15.98:0/2818161781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 msgr2=0x7fc90c104330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 --2- 172.21.15.98:0/2818161781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c104330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc900009910 tx=0x7fc900009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 -- 172.21.15.98:0/2818161781 shutdown_connections 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 --2- 172.21.15.98:0/2818161781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c104330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 -- 172.21.15.98:0/2818161781 >> 172.21.15.98:0/2818161781 conn(0x7fc90c0ff4a0 msgr2=0x7fc90c101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 -- 172.21.15.98:0/2818161781 shutdown_connections 2024-03-16T12:11:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.357+0000 7fc913400700 1 -- 172.21.15.98:0/2818161781 wait complete. 2024-03-16T12:11:01.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.358+0000 7fc913400700 1 Processor -- start 2024-03-16T12:11:01.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.358+0000 7fc913400700 1 -- start start 2024-03-16T12:11:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc913400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc913400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c109f30 con 0x7fc90c103ef0 2024-03-16T12:11:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc910a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc910a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38848/0 (socket says 172.21.15.98:38848) 2024-03-16T12:11:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc910a00700 1 -- 172.21.15.98:0/4191370879 learned_addr learned my addr 172.21.15.98:0/4191370879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.359+0000 7fc910a00700 1 -- 172.21.15.98:0/4191370879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c199b80 con 0x7fc90c103ef0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc910a00700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc900005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900027070 con 0x7fc90c103ef0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc90000be40 con 0x7fc90c103ef0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc90000f460 con 0x7fc90c103ef0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90c199e10 con 0x7fc90c103ef0 2024-03-16T12:11:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.360+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc90c19cb00 con 0x7fc90c103ef0 2024-03-16T12:11:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.362+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90c10a6b0 con 0x7fc90c103ef0 2024-03-16T12:11:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.362+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc90002a070 con 0x7fc90c103ef0 2024-03-16T12:11:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.362+0000 7fc909600700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 0x7fc8f403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.362+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc90001e030 con 0x7fc90c103ef0 2024-03-16T12:11:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.362+0000 7fc90be00700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 0x7fc8f403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.363+0000 7fc90be00700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 0x7fc8f403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc006fd0 tx=0x7fc8fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:01.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.368+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc900013070 con 0x7fc90c103ef0 2024-03-16T12:11:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.593+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fc90c02dc80 con 0x7fc8f4038fc0 2024-03-16T12:11:01.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.599+0000 7fc909600700 1 -- 172.21.15.98:0/4191370879 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc90c02dc80 con 0x7fc8f4038fc0 2024-03-16T12:11:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 msgr2=0x7fc8f403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 0x7fc8f403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc006fd0 tx=0x7fc8fc006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 msgr2=0x7fc90c199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc900005050 comp rx=0 tx=0).stop 2024-03-16T12:11:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 shutdown_connections 2024-03-16T12:11:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f4038fc0 0x7fc8f403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 --2- 172.21.15.98:0/4191370879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c103ef0 0x7fc90c199640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 >> 172.21.15.98:0/4191370879 conn(0x7fc90c0ff4a0 msgr2=0x7fc90c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 shutdown_connections 2024-03-16T12:11:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:01.602+0000 7fc913400700 1 -- 172.21.15.98:0/4191370879 wait complete. 2024-03-16T12:11:02.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:02.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:02.356 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-03-16T12:11:02.356 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:11:02.356 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-03-16T12:11:02.375 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:11:02.375 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:02.432 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-03-16T12:11:02.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch host add smithi153 2024-03-16T12:11:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:03.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:02 smithi098 ceph-mon[31633]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:03.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:02 smithi098 ceph-mon[31633]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T12:11:04.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:03 smithi098 ceph-mon[31633]: Updating smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:04.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:03 smithi098 ceph-mon[31633]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:04.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:03 smithi098 ceph-mon[31633]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.912+0000 7f3858c00700 1 -- 172.21.15.98:0/872306549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 msgr2=0x7f3854104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.912+0000 7f3858c00700 1 --2- 172.21.15.98:0/872306549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f3854104510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3844009910 tx=0x7f3844009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.913+0000 7f3858c00700 1 -- 172.21.15.98:0/872306549 shutdown_connections 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.913+0000 7f3858c00700 1 --2- 172.21.15.98:0/872306549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f3854104510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.913+0000 7f3858c00700 1 -- 172.21.15.98:0/872306549 >> 172.21.15.98:0/872306549 conn(0x7f38540ff9c0 msgr2=0x7f3854101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.913+0000 7f3858c00700 1 -- 172.21.15.98:0/872306549 shutdown_connections 2024-03-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.913+0000 7f3858c00700 1 -- 172.21.15.98:0/872306549 wait complete. 2024-03-16T12:11:04.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.914+0000 7f3858c00700 1 Processor -- start 2024-03-16T12:11:04.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.914+0000 7f3858c00700 1 -- start start 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.914+0000 7f3858c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.914+0000 7f3858c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f385410e500 con 0x7f3854104130 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.914+0000 7f3851e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f3851e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38880/0 (socket says 172.21.15.98:38880) 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f3851e00700 1 -- 172.21.15.98:0/3704406308 learned_addr learned my addr 172.21.15.98:0/3704406308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f3851e00700 1 -- 172.21.15.98:0/3704406308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385419fbb0 con 0x7f3854104130 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f3851e00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3844005850 tx=0x7f3844005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3844026070 con 0x7f3854104130 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f384400bbd0 con 0x7f3854104130 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.915+0000 7f3858c00700 1 -- 172.21.15.98:0/3704406308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385419fea0 con 0x7f3854104130 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.916+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3844003970 con 0x7f3854104130 2024-03-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.916+0000 7f3858c00700 1 -- 172.21.15.98:0/3704406308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38541997f0 con 0x7f3854104130 2024-03-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.917+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3844029070 con 0x7f3854104130 2024-03-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.917+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c005330 con 0x7f3854104130 2024-03-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.917+0000 7f384aa00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 0x7f383803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.917+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f384401e070 con 0x7f3854104130 2024-03-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.918+0000 7f3851400700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 0x7f383803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.919+0000 7f3851400700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 0x7f383803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3840006fd0 tx=0x7f3840006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:04.923+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3844013070 con 0x7f3854104130 2024-03-16T12:11:04.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:04 smithi098 ceph-mon[31633]: Updating smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:04.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:04.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:04.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:05.150+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7f382c000bc0 con 0x7f3838038fc0 2024-03-16T12:11:06.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:05 smithi098 ceph-mon[31633]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:07.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:06 smithi098 ceph-mon[31633]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:08.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:07 smithi098 ceph-mon[31633]: Deploying cephadm binary to smithi153 2024-03-16T12:11:08.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:07 smithi098 ceph-mon[31633]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:09.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.921+0000 7f384aa00700 1 -- 172.21.15.98:0/3704406308 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f382c000bc0 con 0x7f3838038fc0 2024-03-16T12:11:09.909 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 msgr2=0x7f383803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 0x7f383803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3840006fd0 tx=0x7f3840006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 msgr2=0x7f385419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3844005850 tx=0x7f3844005030 comp rx=0 tx=0).stop 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 shutdown_connections 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3838038fc0 0x7f383803b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 --2- 172.21.15.98:0/3704406308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3854104130 0x7f385419f670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.925+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 >> 172.21.15.98:0/3704406308 conn(0x7f38540ff9c0 msgr2=0x7f3854105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:09.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.926+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 shutdown_connections 2024-03-16T12:11:09.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:09.926+0000 7f383fe00700 1 -- 172.21.15.98:0/3704406308 wait complete. 2024-03-16T12:11:10.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:09 smithi098 ceph-mon[31633]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:10.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T12:11:11.165 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:11.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:10 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:11.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:10 smithi098 ceph-mon[31633]: Added host smithi153 2024-03-16T12:11:11.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:10 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:11.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:10 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:11.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:10 smithi098 ceph-mon[31633]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:11.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.337+0000 7f025c400700 1 -- 172.21.15.98:0/1523609458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 msgr2=0x7f0254101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.337+0000 7f025c400700 1 --2- 172.21.15.98:0/1523609458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f0254101030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f024800b0a0 tx=0x7f024800b3b0 comp rx=0 tx=0).stop 2024-03-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.338+0000 7f025c400700 1 -- 172.21.15.98:0/1523609458 shutdown_connections 2024-03-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.338+0000 7f025c400700 1 --2- 172.21.15.98:0/1523609458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f0254101030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.338+0000 7f025c400700 1 -- 172.21.15.98:0/1523609458 >> 172.21.15.98:0/1523609458 conn(0x7f0254077810 msgr2=0x7f02540feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.338+0000 7f025c400700 1 -- 172.21.15.98:0/1523609458 shutdown_connections 2024-03-16T12:11:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.338+0000 7f025c400700 1 -- 172.21.15.98:0/1523609458 wait complete. 2024-03-16T12:11:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f025c400700 1 Processor -- start 2024-03-16T12:11:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f025c400700 1 -- start start 2024-03-16T12:11:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f025c400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f025c400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0254112950 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f0259a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f0259a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47844/0 (socket says 172.21.15.98:47844) 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.339+0000 7f0259a00700 1 -- 172.21.15.98:0/1312202807 learned_addr learned my addr 172.21.15.98:0/1312202807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f0259a00700 1 -- 172.21.15.98:0/1312202807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025419e2d0 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f0259a00700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f80 tx=0x7f024800bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0248020020 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0248018040 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025419e560 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.340+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0248017070 con 0x7f0254100c50 2024-03-16T12:11:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.341+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02541a2660 con 0x7f0254100c50 2024-03-16T12:11:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.342+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0248023020 con 0x7f0254100c50 2024-03-16T12:11:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.342+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0254069fa0 con 0x7f0254100c50 2024-03-16T12:11:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.342+0000 7f0252000700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 0x7f023c03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.342+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f024800f050 con 0x7f0254100c50 2024-03-16T12:11:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.342+0000 7f0259000700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 0x7f023c03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.343+0000 7f0259000700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 0x7f023c03b4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0240006fd0 tx=0x7f0240006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.347+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f024802a020 con 0x7f0254100c50 2024-03-16T12:11:13.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.558+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f025419f0f0 con 0x7f023c039010 2024-03-16T12:11:13.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.559+0000 7f0252000700 1 -- 172.21.15.98:0/1312202807 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f025419f0f0 con 0x7f023c039010 2024-03-16T12:11:13.547 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:11:13.548 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-03-16T12:11:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 msgr2=0x7f023c03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 0x7f023c03b4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0240006fd0 tx=0x7f0240006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 msgr2=0x7f025419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f80 tx=0x7f024800bd20 comp rx=0 tx=0).stop 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 shutdown_connections 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f023c039010 0x7f023c03b4d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 --2- 172.21.15.98:0/1312202807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0254100c50 0x7f025419dd90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 >> 172.21.15.98:0/1312202807 conn(0x7f0254077810 msgr2=0x7f02540fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.562+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 shutdown_connections 2024-03-16T12:11:13.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:13.563+0000 7f025c400700 1 -- 172.21.15.98:0/1312202807 wait complete. 2024-03-16T12:11:13.911 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:13 smithi098 ceph-mon[31633]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:13.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:14.279 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-03-16T12:11:14.280 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:11:14.280 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-03-16T12:11:14.298 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:11:14.298 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:14.356 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-03-16T12:11:14.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch host add smithi173 2024-03-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:14.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:14 smithi098 ceph-mon[31633]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:15.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:15 smithi098 ceph-mon[31633]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 -- 172.21.15.98:0/3049976617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 msgr2=0x7f591c1022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 --2- 172.21.15.98:0/3049976617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c1022e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5910009910 tx=0x7f5910009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 -- 172.21.15.98:0/3049976617 shutdown_connections 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 --2- 172.21.15.98:0/3049976617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c1022e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 -- 172.21.15.98:0/3049976617 >> 172.21.15.98:0/3049976617 conn(0x7f591c0fd7b0 msgr2=0x7f591c0ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 -- 172.21.15.98:0/3049976617 shutdown_connections 2024-03-16T12:11:16.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.991+0000 7f5924200700 1 -- 172.21.15.98:0/3049976617 wait complete. 2024-03-16T12:11:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.992+0000 7f5924200700 1 Processor -- start 2024-03-16T12:11:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.992+0000 7f5924200700 1 -- start start 2024-03-16T12:11:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5924200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5924200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f591c10c2d0 con 0x7f591c101f00 2024-03-16T12:11:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5921800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5921800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58206/0 (socket says 172.21.15.98:58206) 2024-03-16T12:11:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5921800700 1 -- 172.21.15.98:0/2888943703 learned_addr learned my addr 172.21.15.98:0/2888943703 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.993+0000 7f5921800700 1 -- 172.21.15.98:0/2888943703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c19fbd0 con 0x7f591c101f00 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f5921800700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5910006010 tx=0x7f5910005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5910026070 con 0x7f591c101f00 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591000bbd0 con 0x7f591c101f00 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c19fec0 con 0x7f591c101f00 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5910003910 con 0x7f591c101f00 2024-03-16T12:11:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.994+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c199810 con 0x7f591c101f00 2024-03-16T12:11:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.995+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c1033e0 con 0x7f591c101f00 2024-03-16T12:11:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.996+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5910029070 con 0x7f591c101f00 2024-03-16T12:11:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.996+0000 7f591a000700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 0x7f590403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.996+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f591001e070 con 0x7f591c101f00 2024-03-16T12:11:16.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.997+0000 7f5920e00700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 0x7f590403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:16.998+0000 7f5920e00700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 0x7f590403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5908006fd0 tx=0x7f5908006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:17.003+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5910013070 con 0x7f591c101f00 2024-03-16T12:11:16.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:16.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:16.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:16 smithi098 ceph-mon[31633]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:17.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:17.229+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7f591c19ad40 con 0x7f5904039020 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:18.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:19.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:18 smithi098 ceph-mon[31633]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:20.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:19 smithi098 ceph-mon[31633]: Deploying cephadm binary to smithi173 2024-03-16T12:11:20.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:19 smithi098 ceph-mon[31633]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:21.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.964+0000 7f591a000700 1 -- 172.21.15.98:0/2888943703 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f591c19ad40 con 0x7f5904039020 2024-03-16T12:11:21.953 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-03-16T12:11:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 msgr2=0x7f590403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 0x7f590403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5908006fd0 tx=0x7f5908006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 msgr2=0x7f591c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5910006010 tx=0x7f5910005030 comp rx=0 tx=0).stop 2024-03-16T12:11:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 shutdown_connections 2024-03-16T12:11:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5904039020 0x7f590403b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 --2- 172.21.15.98:0/2888943703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f591c101f00 0x7f591c19f690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.968+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 >> 172.21.15.98:0/2888943703 conn(0x7f591c0fd7b0 msgr2=0x7f591c074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.969+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 shutdown_connections 2024-03-16T12:11:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:21.969+0000 7f5924200700 1 -- 172.21.15.98:0/2888943703 wait complete. 2024-03-16T12:11:22.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:21 smithi098 ceph-mon[31633]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:22.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T12:11:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:23.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:23.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:22 smithi098 ceph-mon[31633]: Added host smithi173 2024-03-16T12:11:23.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:23.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:23.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:22 smithi098 ceph-mon[31633]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:24.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:25.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 -- 172.21.15.98:0/2247439723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 msgr2=0x7f4d00104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:25.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 --2- 172.21.15.98:0/2247439723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d00104500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009910 tx=0x7f4cf0009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 -- 172.21.15.98:0/2247439723 shutdown_connections 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 --2- 172.21.15.98:0/2247439723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d00104500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 -- 172.21.15.98:0/2247439723 >> 172.21.15.98:0/2247439723 conn(0x7f4d000ff9b0 msgr2=0x7f4d00101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 -- 172.21.15.98:0/2247439723 shutdown_connections 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.093+0000 7f4d05600700 1 -- 172.21.15.98:0/2247439723 wait complete. 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.094+0000 7f4d05600700 1 Processor -- start 2024-03-16T12:11:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.094+0000 7f4d05600700 1 -- start start 2024-03-16T12:11:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.094+0000 7f4d05600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.094+0000 7f4d05600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0010e4f0 con 0x7f4d00104120 2024-03-16T12:11:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.095+0000 7f4cfea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.095+0000 7f4cfea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58228/0 (socket says 172.21.15.98:58228) 2024-03-16T12:11:25.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.095+0000 7f4cfea00700 1 -- 172.21.15.98:0/3650019298 learned_addr learned my addr 172.21.15.98:0/3650019298 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.095+0000 7f4cfea00700 1 -- 172.21.15.98:0/3650019298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0019fbc0 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.095+0000 7f4cfea00700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0006010 tx=0x7f4cf0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.096+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cf0026070 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.096+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cf000bbd0 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.096+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cf0003910 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.096+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d0019feb0 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.096+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d00199800 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.097+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4cf0029070 con 0x7f4d00104120 2024-03-16T12:11:25.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.097+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d00105600 con 0x7f4d00104120 2024-03-16T12:11:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.097+0000 7f4cf7400700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 0x7f4ce403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.098+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4cf001e070 con 0x7f4d00104120 2024-03-16T12:11:25.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.098+0000 7f4cfe000700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 0x7f4ce403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:25.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.098+0000 7f4cfe000700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 0x7f4ce403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4cec006fd0 tx=0x7f4cec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:25.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.103+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cf0013070 con 0x7f4d00104120 2024-03-16T12:11:25.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.334+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d0019ad30 con 0x7f4ce4038fc0 2024-03-16T12:11:25.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.336+0000 7f4cf7400700 1 -- 172.21.15.98:0/3650019298 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f4d0019ad30 con 0x7f4ce4038fc0 2024-03-16T12:11:25.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:11:25.324 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 msgr2=0x7f4ce403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 0x7f4ce403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4cec006fd0 tx=0x7f4cec006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 msgr2=0x7f4d0019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0006010 tx=0x7f4cf0005030 comp rx=0 tx=0).stop 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 shutdown_connections 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4ce4038fc0 0x7f4ce403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 --2- 172.21.15.98:0/3650019298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d00104120 0x7f4d0019f680 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:25.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 >> 172.21.15.98:0/3650019298 conn(0x7f4d000ff9b0 msgr2=0x7f4d00106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:25.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 shutdown_connections 2024-03-16T12:11:25.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:25.339+0000 7f4d05600700 1 -- 172.21.15.98:0/3650019298 wait complete. 2024-03-16T12:11:25.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:25 smithi098 ceph-mon[31633]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:25.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:26.058 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T12:11:26.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T12:11:26.484 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:26.911 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:26 smithi098 ceph-mon[31633]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:27.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:27 smithi098 ceph-mon[31633]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 -- 172.21.15.98:0/2253172692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 msgr2=0x7ff7c41022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 --2- 172.21.15.98:0/2253172692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c41022e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009910 tx=0x7ff7b4009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 -- 172.21.15.98:0/2253172692 shutdown_connections 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 --2- 172.21.15.98:0/2253172692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c41022e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 -- 172.21.15.98:0/2253172692 >> 172.21.15.98:0/2253172692 conn(0x7ff7c40fd7b0 msgr2=0x7ff7c40ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 -- 172.21.15.98:0/2253172692 shutdown_connections 2024-03-16T12:11:28.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.600+0000 7ff7ca200700 1 -- 172.21.15.98:0/2253172692 wait complete. 2024-03-16T12:11:28.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.601+0000 7ff7ca200700 1 Processor -- start 2024-03-16T12:11:28.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7ca200700 1 -- start start 2024-03-16T12:11:28.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7ca200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:28.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7ca200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c410c2d0 con 0x7ff7c4101f00 2024-03-16T12:11:28.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:28.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7c3400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54078/0 (socket says 172.21.15.98:54078) 2024-03-16T12:11:28.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.602+0000 7ff7c3400700 1 -- 172.21.15.98:0/1359130386 learned_addr learned my addr 172.21.15.98:0/1359130386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:28.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7c3400700 1 -- 172.21.15.98:0/1359130386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c40764c0 con 0x7ff7c4101f00 2024-03-16T12:11:28.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7c3400700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006010 tx=0x7ff7b4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:28.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7b4026070 con 0x7ff7c4101f00 2024-03-16T12:11:28.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b400bc30 con 0x7ff7c4101f00 2024-03-16T12:11:28.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c4074b70 con 0x7ff7c4101f00 2024-03-16T12:11:28.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7b40039d0 con 0x7ff7c4101f00 2024-03-16T12:11:28.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.603+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c4075030 con 0x7ff7c4101f00 2024-03-16T12:11:28.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.605+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c41033e0 con 0x7ff7c4101f00 2024-03-16T12:11:28.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.605+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff7b4029070 con 0x7ff7c4101f00 2024-03-16T12:11:28.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.605+0000 7ff7bbe00700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 0x7ff7a803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:28.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.605+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff7b401e070 con 0x7ff7c4101f00 2024-03-16T12:11:28.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.606+0000 7ff7c2a00700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 0x7ff7a803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:28.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.606+0000 7ff7c2a00700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 0x7ff7a803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0006fd0 tx=0x7ff7b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:28.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.611+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7b4013070 con 0x7ff7c4101f00 2024-03-16T12:11:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.842+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff7c4075ba0 con 0x7ff7c4101f00 2024-03-16T12:11:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.911+0000 7ff7bbe00700 1 -- 172.21.15.98:0/1359130386 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7ff7b402bd20 con 0x7ff7c4101f00 2024-03-16T12:11:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.915+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 msgr2=0x7ff7a803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.915+0000 7ff7ca200700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 0x7ff7a803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0006fd0 tx=0x7ff7b0006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.915+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 msgr2=0x7ff7c40745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.915+0000 7ff7ca200700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006010 tx=0x7ff7b4005030 comp rx=0 tx=0).stop 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 shutdown_connections 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7a8039020 0x7ff7a803b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 --2- 172.21.15.98:0/1359130386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7c4101f00 0x7ff7c40745d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 >> 172.21.15.98:0/1359130386 conn(0x7ff7c40fd7b0 msgr2=0x7ff7c41038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 shutdown_connections 2024-03-16T12:11:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:28.916+0000 7ff7ca200700 1 -- 172.21.15.98:0/1359130386 wait complete. 2024-03-16T12:11:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-03-16T12:11:29.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:29.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:28 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T12:11:29.643 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-03-16T12:11:29.643 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-03-16T12:11:29.643 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-03-16T12:11:29.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c' 2024-03-16T12:11:29.868 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:30.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:30.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:30.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 -- 172.21.15.173:0/340912480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 msgr2=0x7f45081080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 --2- 172.21.15.173:0/340912480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081080d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009910 tx=0x7f44fc009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 -- 172.21.15.173:0/340912480 shutdown_connections 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 --2- 172.21.15.173:0/340912480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081080d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 -- 172.21.15.173:0/340912480 >> 172.21.15.173:0/340912480 conn(0x7f45080781d0 msgr2=0x7f45080785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 -- 172.21.15.173:0/340912480 shutdown_connections 2024-03-16T12:11:31.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.397+0000 7f4510121700 1 -- 172.21.15.173:0/340912480 wait complete. 2024-03-16T12:11:31.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f4510121700 1 Processor -- start 2024-03-16T12:11:31.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f4510121700 1 -- start start 2024-03-16T12:11:31.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f4510121700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:31.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f4510121700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f450810c310 con 0x7f4508107cf0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f450debd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f450debd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.173:59704/0 (socket says 172.21.15.173:59704) 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.398+0000 7f450debd700 1 -- 172.21.15.173:0/3821441090 learned_addr learned my addr 172.21.15.173:0/3821441090 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.399+0000 7f450debd700 1 -- 172.21.15.173:0/3821441090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4508103490 con 0x7f4508107cf0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.399+0000 7f450debd700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44fc006010 tx=0x7f44fc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.399+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44fc026070 con 0x7f4508107cf0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.399+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4508101b40 con 0x7f4508107cf0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.400+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4508102000 con 0x7f4508107cf0 2024-03-16T12:11:31.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.400+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44fc00bbd0 con 0x7f4508107cf0 2024-03-16T12:11:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.400+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44fc0038e0 con 0x7f4508107cf0 2024-03-16T12:11:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.400+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4508069e60 con 0x7f4508107cf0 2024-03-16T12:11:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.401+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f44fc029070 con 0x7f4508107cf0 2024-03-16T12:11:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.401+0000 7f44faffd700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 0x7f44f403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.401+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f44fc01e070 con 0x7f4508107cf0 2024-03-16T12:11:31.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.401+0000 7f450d6bc700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 0x7f44f403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:31.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.402+0000 7f450d6bc700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 0x7f44f403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:31.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.415+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44fc013070 con 0x7f4508107cf0 2024-03-16T12:11:31.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.621+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}) -- 0x7f4508102b40 con 0x7f44f4038fc0 2024-03-16T12:11:31.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.636+0000 7f44faffd700 1 -- 172.21.15.173:0/3821441090 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4508102b40 con 0x7f44f4038fc0 2024-03-16T12:11:31.623 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-03-16T12:11:31.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 msgr2=0x7f44f403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:31.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 0x7f44f403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:31.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 msgr2=0x7f45081015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:31.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44fc006010 tx=0x7f44fc005030 comp rx=0 tx=0).stop 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 shutdown_connections 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44f4038fc0 0x7f44f403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 --2- 172.21.15.173:0/3821441090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4508107cf0 0x7f45081015a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 >> 172.21.15.173:0/3821441090 conn(0x7f45080781d0 msgr2=0x7f45080fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 shutdown_connections 2024-03-16T12:11:31.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:31.638+0000 7f4510121700 1 -- 172.21.15.173:0/3821441090 wait complete. 2024-03-16T12:11:32.214 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T12:11:32.217 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T12:11:32.219 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T12:11:32.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T12:11:32.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:32.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:32.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:32.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:32.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:32.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:32.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:32.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:33.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:32 smithi098 ceph-mon[31633]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:33.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:32 smithi098 ceph-mon[31633]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c;count:3 2024-03-16T12:11:33.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:32 smithi098 ceph-mon[31633]: Deploying daemon mon.c on smithi173 2024-03-16T12:11:33.302 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:34.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:33 smithi098 ceph-mon[31633]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:35.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:34 smithi098 ceph-mon[31633]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 -- 172.21.15.173:0/2754781606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 msgr2=0x7f82081011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 --2- 172.21.15.173:0/2754781606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f82081011e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009910 tx=0x7f81f8009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 -- 172.21.15.173:0/2754781606 shutdown_connections 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 --2- 172.21.15.173:0/2754781606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f82081011e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 -- 172.21.15.173:0/2754781606 >> 172.21.15.173:0/2754781606 conn(0x7f8208077810 msgr2=0x7f82080feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 -- 172.21.15.173:0/2754781606 shutdown_connections 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.916+0000 7f820ccad700 1 -- 172.21.15.173:0/2754781606 wait complete. 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.917+0000 7f820ccad700 1 Processor -- start 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.917+0000 7f820ccad700 1 -- start start 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.917+0000 7f820ccad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.917+0000 7f820ccad700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f820810b1d0 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.918+0000 7f820659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.918+0000 7f820659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.173:55714/0 (socket says 172.21.15.173:55714) 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.918+0000 7f820659c700 1 -- 172.21.15.173:0/371746922 learned_addr learned my addr 172.21.15.173:0/371746922 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.918+0000 7f820659c700 1 -- 172.21.15.173:0/371746922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820806a4c0 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f820659c700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f81f8005850 tx=0x7f81f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f8026070 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8208196980 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8208196e40 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f800bbd0 con 0x7f8208100e00 2024-03-16T12:11:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.919+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f8003680 con 0x7f8208100e00 2024-03-16T12:11:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.920+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e8005330 con 0x7f8208100e00 2024-03-16T12:11:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.920+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f81f8029070 con 0x7f8208100e00 2024-03-16T12:11:35.909 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.921+0000 7f81f77fe700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 0x7f81f003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:35.909 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.921+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f81f801e070 con 0x7f8208100e00 2024-03-16T12:11:35.909 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.921+0000 7f8205d9b700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 0x7f81f003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:35.909 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.922+0000 7f8205d9b700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 0x7f81f003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81fc006fd0 tx=0x7f81fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:35.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:35.936+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81f8013070 con 0x7f8208100e00 2024-03-16T12:11:36.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.226+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f81e80058f0 con 0x7f8208100e00 2024-03-16T12:11:36.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.227+0000 7f81f77fe700 1 -- 172.21.15.173:0/371746922 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f81f8034090 con 0x7f8208100e00 2024-03-16T12:11:36.215 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:11:36.215 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","modified":"2024-03-16T12:09:20.978440Z","created":"2024-03-16T12:09:20.978440Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 msgr2=0x7f81f003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 0x7f81f003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81fc006fd0 tx=0x7f81fc006e40 comp rx=0 tx=0).stop 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 msgr2=0x7f8208196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f81f8005850 tx=0x7f81f8005030 comp rx=0 tx=0).stop 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 shutdown_connections 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81f0038fc0 0x7f81f003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 --2- 172.21.15.173:0/371746922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8208100e00 0x7f8208196440 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 >> 172.21.15.173:0/371746922 conn(0x7f8208077810 msgr2=0x7f82081027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:36.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 shutdown_connections 2024-03-16T12:11:36.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:36.229+0000 7f820ccad700 1 -- 172.21.15.173:0/371746922 wait complete. 2024-03-16T12:11:36.218 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-03-16T12:11:36.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 podman[36599]: 2024-03-16 12:11:36.323098669 +0000 UTC m=+0.342288020 container start 7eb0a02b24449406021362f4813dd597a6bb7f079d4d2d28207a63e5b2fbe6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:11:36.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:36 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/371746922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:36.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T12:11:36.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T12:11:36.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pidfile_write: ignore empty --pid-file 2024-03-16T12:11:36.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: load: jerasure load: lrc 2024-03-16T12:11:36.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: RocksDB version: 7.9.2 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Git sha 0 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: DB SUMMARY 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: DB Session ID: W5X1CPE8W1XZ46MD9MOP 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: CURRENT file: CURRENT 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.error_if_exists: 0 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.create_if_missing: 0 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.paranoid_checks: 1 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T12:11:36.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.env: 0x55f01ab1dc60 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.info_log: 0x55f01c403ac0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.statistics: (nil) 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.use_fsync: 0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_log_file_size: 0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_fallocate: 1 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T12:11:36.828 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.use_direct_reads: 0 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.db_log_dir: 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.wal_dir: 2024-03-16T12:11:36.829 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.write_buffer_manager: 0x55f01c3d61e0 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T12:11:36.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.unordered_write: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.row_cache: None 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.wal_filter: None 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.two_write_queues: 0 2024-03-16T12:11:36.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.wal_compression: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.atomic_flush: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.log_readahead_size: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_background_jobs: 2 2024-03-16T12:11:36.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_background_compactions: -1 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_subcompactions: 1 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_open_files: -1 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T12:11:36.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_background_flushes: -1 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Compression algorithms supported: 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kZSTD supported: 0 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kXpressCompression supported: 0 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kZlibCompression supported: 1 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kSnappyCompression supported: 1 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kLZ4Compression supported: 1 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: kBZip2Compression supported: 0 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T12:11:36.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.merge_operator: 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_filter: None 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f01c403bc0) 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_type: 0 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_index_type: 0 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_shortening: 1 2024-03-16T12:11:36.836 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: checksum: 4 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: no_block_cache: 0 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache: 0x55f01c409090 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_options: 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: capacity : 536870912 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_shard_bits : 4 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: strict_capacity_limit : 0 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_compressed: (nil) 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: persistent_cache: (nil) 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size: 4096 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size_deviation: 10 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_restart_interval: 16 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_block_restart_interval: 1 2024-03-16T12:11:36.837 INFO:journalctl@ceph.mon.c.smithi173.stdout: metadata_block_size: 4096 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: partition_filters: 0 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: use_delta_encoding: 1 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: filter_policy: bloomfilter 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: whole_key_filtering: 1 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: verify_compression: 0 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: format_version: 5 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_index_compression: 1 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_align: 0 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: max_auto_readahead_size: 262144 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: prepopulate_block_cache: 0 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: initial_auto_readahead_size: 8192 2024-03-16T12:11:36.838 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression: NoCompression 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.num_levels: 7 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:11:36.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T12:11:36.840 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T12:11:36.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T12:11:36.842 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.table_properties_collectors: 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.inplace_update_support: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.bloom_locality: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.max_successive_merges: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.ttl: 2592000 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T12:11:36.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enable_blob_files: false 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.min_blob_size: 0 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2ab0ea95-cc58-4e5c-85c2-1a7c99902ca7 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591096352988, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T12:11:36.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591096353573, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710591096, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2ab0ea95-cc58-4e5c-85c2-1a7c99902ca7", "db_session_id": "W5X1CPE8W1XZ46MD9MOP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591096353677, "job": 1, "event": "recovery_finished"} 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f01c4fc000 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: DB pointer 0x55f01c4e6000 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** DB Stats ** 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:11:36.845 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T12:11:36.846 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache BinnedLRUCache@0x55f01c409090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T12:11:36.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: starting mon.c rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(???) e0 preinit fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: e1 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: legacy client fscid: -1 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout: No filesystems configured 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T12:11:36.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Deploying cephadm binary to smithi153 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Added host smithi153 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:36.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Deploying cephadm binary to smithi173 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Added host smithi173 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:36.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:36.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c;count:3 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: Deploying daemon mon.c on smithi173 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/371746922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:36.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 ceph-mon[36632]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T12:11:37.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 bash[36599]: 7eb0a02b24449406021362f4813dd597a6bb7f079d4d2d28207a63e5b2fbe6b3 2024-03-16T12:11:37.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:36 smithi173 systemd[1]: Started Ceph mon.c for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:11:37.856 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T12:11:37.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T12:11:38.100 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:11:41.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:41.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: mon.a calling monitor election 2024-03-16T12:11:41.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: mon.c calling monitor election 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: monmap epoch 2 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: last_changed 2024-03-16T12:11:36.387657+0000 2024-03-16T12:11:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: min_mon_release 19 (squid) 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: election_strategy: 1 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: fsmap 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: mgrmap e13: x(active, since 58s) 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: overall HEALTH_OK 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: mon.a calling monitor election 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: mon.c calling monitor election 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:41.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: monmap epoch 2 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: last_changed 2024-03-16T12:11:36.387657+0000 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: min_mon_release 19 (squid) 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: election_strategy: 1 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: fsmap 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: mgrmap e13: x(active, since 58s) 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: overall HEALTH_OK 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:41.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:42.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 -- 172.21.15.173:0/3086188803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 msgr2=0x7f3b64005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 --2- 172.21.15.173:0/3086188803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b64005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009bf0 tx=0x7f3b70009f00 comp rx=0 tx=0).stop 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 -- 172.21.15.173:0/3086188803 shutdown_connections 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 --2- 172.21.15.173:0/3086188803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b64005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 -- 172.21.15.173:0/3086188803 >> 172.21.15.173:0/3086188803 conn(0x7f3b780fc780 msgr2=0x7f3b780feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 -- 172.21.15.173:0/3086188803 shutdown_connections 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.525+0000 7f3b7f453700 1 -- 172.21.15.173:0/3086188803 wait complete. 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 Processor -- start 2024-03-16T12:11:42.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 -- start start 2024-03-16T12:11:42.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:42.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:42.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:42.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:42 smithi173 ceph-mon[36632]: Deploying daemon mon.b on smithi153 2024-03-16T12:11:42.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b7819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7810c020 con 0x7f3b78100b60 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.526+0000 7f3b7f453700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7810bea0 con 0x7f3b7819fcf0 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:55186/0 (socket says 172.21.15.173:55186) 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 -- 172.21.15.173:0/573473521 learned_addr learned my addr 172.21.15.173:0/573473521 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 -- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 msgr2=0x7f3b78199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 -- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 msgr2=0x7f3b78199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7c9ee700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:11:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7d1ef700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b7819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7d1ef700 1 -- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 msgr2=0x7f3b78199970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7d1ef700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.527+0000 7f3b7d1ef700 1 -- 172.21.15.173:0/573473521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781a0330 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b7d1ef700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b7819f7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b70007950 tx=0x7f3b70003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3b70004c40 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b78199eb0 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b70004de0 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3b70005210 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.528+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7819a440 con 0x7f3b78100b60 2024-03-16T12:11:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.529+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7806a480 con 0x7f3b78100b60 2024-03-16T12:11:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.530+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3b7001e070 con 0x7f3b78100b60 2024-03-16T12:11:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.530+0000 7f3b6e7fc700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 0x7f3b5c03b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.530+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3b7001c070 con 0x7f3b78100b60 2024-03-16T12:11:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.530+0000 7f3b7c9ee700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 0x7f3b5c03b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.531+0000 7f3b7c9ee700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 0x7f3b5c03b1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009180 tx=0x7f3b68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:42.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.546+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b7001edc0 con 0x7f3b78100b60 2024-03-16T12:11:42.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:42.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:42.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:42.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:42 smithi098 ceph-mon[31633]: Deploying daemon mon.b on smithi153 2024-03-16T12:11:42.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:42.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.818+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3b7819aea0 con 0x7f3b78100b60 2024-03-16T12:11:42.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.819+0000 7f3b6e7fc700 1 -- 172.21.15.173:0/573473521 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f3b70037090 con 0x7f3b78100b60 2024-03-16T12:11:42.807 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:11:42.807 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","modified":"2024-03-16T12:11:36.387657Z","created":"2024-03-16T12:09:20.978440Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 msgr2=0x7f3b5c03b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 0x7f3b5c03b1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009180 tx=0x7f3b68008040 comp rx=0 tx=0).stop 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 msgr2=0x7f3b7819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b7819f7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b70007950 tx=0x7f3b70003e00 comp rx=0 tx=0).stop 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 shutdown_connections 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3b5c038d10 0x7f3b5c03b1d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b78100b60 0x7f3b7819f7b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:42.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 --2- 172.21.15.173:0/573473521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b7819fcf0 0x7f3b78199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:42.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 >> 172.21.15.173:0/573473521 conn(0x7f3b780fc780 msgr2=0x7f3b780fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:42.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 shutdown_connections 2024-03-16T12:11:42.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:42.821+0000 7f3b7f453700 1 -- 172.21.15.173:0/573473521 wait complete. 2024-03-16T12:11:42.810 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-03-16T12:11:43.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:43 smithi098 ceph-mon[31633]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:43.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:43 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/573473521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:43.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:43 smithi173 ceph-mon[36632]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:43.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:43 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/573473521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:44.360 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T12:11:44.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T12:11:44.603 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:11:45.090 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:44 smithi153 podman[36882]: 2024-03-16 12:11:44.997188335 +0000 UTC m=+0.118818901 container create 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 podman[36882]: 2024-03-16 12:11:45.191875512 +0000 UTC m=+0.313506097 container init 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 podman[36882]: 2024-03-16 12:11:45.195101906 +0000 UTC m=+0.316732478 container start 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pidfile_write: ignore empty --pid-file 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: load: jerasure load: lrc 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: RocksDB version: 7.9.2 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Git sha 0 2024-03-16T12:11:45.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: DB SUMMARY 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: DB Session ID: JHNJBL3VIMADM5OVLVA2 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: CURRENT file: CURRENT 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.error_if_exists: 0 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.create_if_missing: 0 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.paranoid_checks: 1 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.env: 0x55d3ece6dc60 2024-03-16T12:11:45.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T12:11:45.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.info_log: 0x55d3ee195ac0 2024-03-16T12:11:45.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T12:11:45.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.statistics: (nil) 2024-03-16T12:11:45.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.use_fsync: 0 2024-03-16T12:11:45.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_log_file_size: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_fallocate: 1 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.use_direct_reads: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.db_log_dir: 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.wal_dir: 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T12:11:45.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.write_buffer_manager: 0x55d3ee1681e0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T12:11:45.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.unordered_write: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.row_cache: None 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.wal_filter: None 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.two_write_queues: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.wal_compression: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.atomic_flush: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T12:11:45.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.log_readahead_size: 0 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_background_jobs: 2 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_background_compactions: -1 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_subcompactions: 1 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T12:11:45.516 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_open_files: -1 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_background_flushes: -1 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Compression algorithms supported: 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kZSTD supported: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kXpressCompression supported: 0 2024-03-16T12:11:45.517 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kZlibCompression supported: 1 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kSnappyCompression supported: 1 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kLZ4Compression supported: 1 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: kBZip2Compression supported: 0 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.merge_operator: 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_filter: None 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T12:11:45.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d3ee195bc0) 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_type: 0 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_index_type: 0 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_shortening: 1 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: checksum: 4 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: no_block_cache: 0 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache: 0x55d3ee19b090 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-03-16T12:11:45.519 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_options: 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: capacity : 536870912 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_shard_bits : 4 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: strict_capacity_limit : 0 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_compressed: (nil) 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: persistent_cache: (nil) 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size: 4096 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size_deviation: 10 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_restart_interval: 16 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_block_restart_interval: 1 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: metadata_block_size: 4096 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: partition_filters: 0 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: use_delta_encoding: 1 2024-03-16T12:11:45.520 INFO:journalctl@ceph.mon.b.smithi153.stdout: filter_policy: bloomfilter 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: whole_key_filtering: 1 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: verify_compression: 0 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: format_version: 5 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_index_compression: 1 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_align: 0 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: max_auto_readahead_size: 262144 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: prepopulate_block_cache: 0 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: initial_auto_readahead_size: 8192 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression: NoCompression 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T12:11:45.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.num_levels: 7 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:11:45.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T12:11:45.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T12:11:45.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T12:11:45.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T12:11:45.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T12:11:45.525 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.table_properties_collectors: 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.inplace_update_support: 0 2024-03-16T12:11:45.526 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.bloom_locality: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.max_successive_merges: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.ttl: 2592000 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enable_blob_files: false 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.min_blob_size: 0 2024-03-16T12:11:45.527 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b8e3cf68-6c66-4749-98ac-01dfe3a7ab4d 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591105224369, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591105224785, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710591105, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b8e3cf68-6c66-4749-98ac-01dfe3a7ab4d", "db_session_id": "JHNJBL3VIMADM5OVLVA2", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710591105224871, "job": 1, "event": "recovery_finished"} 2024-03-16T12:11:45.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d3ee28e000 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: DB pointer 0x55d3ee278000 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** DB Stats ** 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-16T12:11:45.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache BinnedLRUCache@0x55d3ee19b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: starting mon.b rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:45.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(???) e0 preinit fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).mds e1 new map 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: e1 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: legacy client fscid: -1 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: No filesystems configured 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T12:11:45.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Deploying cephadm binary to smithi153 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Added host smithi153 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.534 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Deploying cephadm binary to smithi173 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Added host smithi173 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:11:45.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1359130386' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c;count:3 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Deploying daemon mon.c on smithi173 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/371746922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.a calling monitor election 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.c calling monitor election 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: monmap epoch 2 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: last_changed 2024-03-16T12:11:36.387657+0000 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: min_mon_release 19 (squid) 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: election_strategy: 1 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: fsmap 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mgrmap e13: x(active, since 58s) 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: overall HEALTH_OK 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: Deploying daemon mon.b on smithi153 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/573473521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 ceph-mon[36901]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 bash[36882]: 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d 2024-03-16T12:11:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:45 smithi153 systemd[1]: Started Ceph mon.b for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 -- 172.21.15.173:0/647106758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd00035c0 msgr2=0x7f9cd0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 --2- 172.21.15.173:0/647106758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd00035c0 0x7f9cd0005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ce806a590 tx=0x7f9cd800fe60 comp rx=0 tx=0).stop 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 -- 172.21.15.173:0/647106758 shutdown_connections 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 --2- 172.21.15.173:0/647106758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd00035c0 0x7f9cd0005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 -- 172.21.15.173:0/647106758 >> 172.21.15.173:0/647106758 conn(0x7f9ce80fc7c0 msgr2=0x7f9ce80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 -- 172.21.15.173:0/647106758 shutdown_connections 2024-03-16T12:11:50.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.336+0000 7f9cee974700 1 -- 172.21.15.173:0/647106758 wait complete. 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.337+0000 7f9cee974700 1 Processor -- start 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.337+0000 7f9cee974700 1 -- start start 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 0x7f9ce8198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8109e00 con 0x7f9ce8197e50 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8109c80 con 0x7f9ce819dc90 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cee974700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8109f80 con 0x7f9cd00035c0 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:56370/0 (socket says 172.21.15.173:56370) 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 -- 172.21.15.173:0/1181769296 learned_addr learned my addr 172.21.15.173:0/1181769296 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:11:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 msgr2=0x7f9ce819d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 msgr2=0x7f9ce819d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce77fe700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce7fff700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9cecf11700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 0x7f9ce8198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce77fe700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 msgr2=0x7f9ce8197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce77fe700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 msgr2=0x7f9ce8197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.338+0000 7f9ce77fe700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T12:11:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9ce77fe700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T12:11:50.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9cecf11700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 msgr2=0x7f9ce819d750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:11:50.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9cecf11700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9cecf11700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 msgr2=0x7f9ce8197910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T12:11:50.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9cecf11700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.339+0000 7f9cecf11700 1 -- 172.21.15.173:0/1181769296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce81a4620 con 0x7f9ce8197e50 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.340+0000 7f9cecf11700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 0x7f9ce8198300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000bd50 tx=0x7f9ce000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.341+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0019070 con 0x7f9ce8197e50 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.341+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce81a4850 con 0x7f9ce8197e50 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.341+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce000cd10 con 0x7f9ce8197e50 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.341+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00127e0 con 0x7f9ce8197e50 2024-03-16T12:11:50.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.341+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce81a4d90 con 0x7f9ce8197e50 2024-03-16T12:11:50.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.342+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce8063900 con 0x7f9ce8197e50 2024-03-16T12:11:50.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.342+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9ce001c070 con 0x7f9ce8197e50 2024-03-16T12:11:50.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.343+0000 7f9ce57fa700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 0x7f9cd403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:50.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.343+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9ce004b8e0 con 0x7f9ce8197e50 2024-03-16T12:11:50.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.343+0000 7f9ce7fff700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 0x7f9cd403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:50.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.344+0000 7f9ce7fff700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 0x7f9cd403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800fa10 tx=0x7f9cd8005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:50.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.362+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ce004a2f0 con 0x7f9ce8197e50 2024-03-16T12:11:50.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: mon.c calling monitor election 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: mon.a calling monitor election 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:50.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: monmap epoch 3 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: min_mon_release 19 (squid) 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: election_strategy: 1 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: fsmap 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: mgrmap e13: x(active, since 67s) 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T12:11:50.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: mon.b (rank 2) addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] is down (out of quorum) 2024-03-16T12:11:50.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:50.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:50.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.643+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9ce81a5050 con 0x7f9ce8197e50 2024-03-16T12:11:50.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.644+0000 7f9ce57fa700 1 -- 172.21.15.173:0/1181769296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f9ce004a670 con 0x7f9ce8197e50 2024-03-16T12:11:50.631 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-16T12:11:50.631 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","modified":"2024-03-16T12:11:45.247026Z","created":"2024-03-16T12:09:20.978440Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T12:11:50.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 msgr2=0x7f9cd403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:50.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 0x7f9cd403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800fa10 tx=0x7f9cd8005a90 comp rx=0 tx=0).stop 2024-03-16T12:11:50.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 msgr2=0x7f9ce8198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 0x7f9ce8198300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000bd50 tx=0x7f9ce000be30 comp rx=0 tx=0).stop 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 shutdown_connections 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ce8197e50 0x7f9ce8198300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cd00035c0 0x7f9ce819d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9cd4039450 0x7f9cd403b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 --2- 172.21.15.173:0/1181769296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce819dc90 0x7f9ce8197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 >> 172.21.15.173:0/1181769296 conn(0x7f9ce80fc7c0 msgr2=0x7f9ce8103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 shutdown_connections 2024-03-16T12:11:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:50.646+0000 7f9cee974700 1 -- 172.21.15.173:0/1181769296 wait complete. 2024-03-16T12:11:50.635 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-03-16T12:11:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: mon.c calling monitor election 2024-03-16T12:11:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: mon.a calling monitor election 2024-03-16T12:11:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: monmap epoch 3 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:50.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: min_mon_release 19 (squid) 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: election_strategy: 1 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: fsmap 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: mgrmap e13: x(active, since 67s) 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: mon.b (rank 2) addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] is down (out of quorum) 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:50.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:51.199 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T12:11:51.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T12:11:51.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:51.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:51.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:51.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/1181769296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:51.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:51.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/1181769296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:51.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: mon.c calling monitor election 2024-03-16T12:11:52.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: mon.a calling monitor election 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: monmap epoch 3 2024-03-16T12:11:52.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: min_mon_release 19 (squid) 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: election_strategy: 1 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: fsmap 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: mgrmap e13: x(active, since 67s) 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:11:52.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: mon.b (rank 2) addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] is down (out of quorum) 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/1181769296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:52.512 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:53.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mon.b calling monitor election 2024-03-16T12:11:53.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mon.b calling monitor election 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mon.c calling monitor election 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mon.a calling monitor election 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: monmap epoch 3 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:53.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: min_mon_release 19 (squid) 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: election_strategy: 1 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: fsmap 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: mgrmap e13: x(active, since 69s) 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: overall HEALTH_OK 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: Updating smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:53.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mon.b calling monitor election 2024-03-16T12:11:53.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mon.b calling monitor election 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mon.c calling monitor election 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mon.a calling monitor election 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: monmap epoch 3 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: min_mon_release 19 (squid) 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: election_strategy: 1 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:53.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: fsmap 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: mgrmap e13: x(active, since 69s) 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: overall HEALTH_OK 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: Updating smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:53.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mon.b calling monitor election 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mon.b calling monitor election 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mon.c calling monitor election 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mon.a calling monitor election 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: monmap epoch 3 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: last_changed 2024-03-16T12:11:45.247026+0000 2024-03-16T12:11:53.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: created 2024-03-16T12:09:20.978440+0000 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: min_mon_release 19 (squid) 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: election_strategy: 1 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: fsmap 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: mgrmap e13: x(active, since 69s) 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: overall HEALTH_OK 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: Updating smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T12:11:53.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:53.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.730+0000 7f6050800700 1 -- 172.21.15.98:0/2914502141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 msgr2=0x7f6048103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.730+0000 7f6050800700 1 --2- 172.21.15.98:0/2914502141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f6048103240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f603c009910 tx=0x7f603c009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.731+0000 7f6050800700 1 -- 172.21.15.98:0/2914502141 shutdown_connections 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.731+0000 7f6050800700 1 --2- 172.21.15.98:0/2914502141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f6048103240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.731+0000 7f6050800700 1 -- 172.21.15.98:0/2914502141 >> 172.21.15.98:0/2914502141 conn(0x7f6048077810 msgr2=0x7f60480feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.731+0000 7f6050800700 1 -- 172.21.15.98:0/2914502141 shutdown_connections 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.731+0000 7f6050800700 1 -- 172.21.15.98:0/2914502141 wait complete. 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.732+0000 7f6050800700 1 Processor -- start 2024-03-16T12:11:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.732+0000 7f6050800700 1 -- start start 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604819fe60 0x7f6048199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604819a020 0x7f604819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f604810c350 con 0x7f6048102e60 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f604810c1d0 con 0x7f604819a020 2024-03-16T12:11:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f6050800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f604810c4d0 con 0x7f604819fe60 2024-03-16T12:11:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f604de00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f604de00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60772/0 (socket says 172.21.15.98:60772) 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f604de00700 1 -- 172.21.15.98:0/4293210452 learned_addr learned my addr 172.21.15.98:0/4293210452 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f604d400700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604819fe60 0x7f6048199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.733+0000 7f604e800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604819a020 0x7f604819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 -- 172.21.15.98:0/4293210452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604819fe60 msgr2=0x7f6048199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604819fe60 0x7f6048199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 -- 172.21.15.98:0/4293210452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604819a020 msgr2=0x7f604819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604819a020 0x7f604819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 -- 172.21.15.98:0/4293210452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60481a67f0 con 0x7f6048102e60 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f604de00700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f603c0092e0 tx=0x7f603c004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c026070 con 0x7f6048102e60 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60481a69c0 con 0x7f6048102e60 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f603c00b750 con 0x7f6048102e60 2024-03-16T12:11:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.734+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c003a00 con 0x7f6048102e60 2024-03-16T12:11:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.735+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a6ea0 con 0x7f6048102e60 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.736+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f603c029070 con 0x7f6048102e60 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.736+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6048063900 con 0x7f6048102e60 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.736+0000 7f6042a00700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 0x7f603003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.736+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f603c04cc10 con 0x7f6048102e60 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.737+0000 7f604d400700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 0x7f603003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.737+0000 7f604d400700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 0x7f603003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6038009ad0 tx=0x7f6038009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.747+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f603c013070 con 0x7f6048102e60 2024-03-16T12:11:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.970+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f60481a70a0 con 0x7f6048102e60 2024-03-16T12:11:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.971+0000 7f6042a00700 1 -- 172.21.15.98:0/4293210452 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f603c030020 con 0x7f6048102e60 2024-03-16T12:11:53.958 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:53.958 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-16T12:11:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:11:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-03-16T12:11:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 msgr2=0x7f603003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 0x7f603003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6038009ad0 tx=0x7f6038009210 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 msgr2=0x7f604819f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f603c0092e0 tx=0x7f603c004fa0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 shutdown_connections 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6030039450 0x7f603003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.975+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604819a020 0x7f604819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.976+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6048102e60 0x7f604819f920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.976+0000 7f6050800700 1 --2- 172.21.15.98:0/4293210452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f604819fe60 0x7f6048199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.976+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 >> 172.21.15.98:0/4293210452 conn(0x7f6048077810 msgr2=0x7f60481080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.976+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 shutdown_connections 2024-03-16T12:11:53.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:11:53.976+0000 7f6050800700 1 -- 172.21.15.98:0/4293210452 wait complete. 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: Reconfiguring daemon mon.a on smithi098 2024-03-16T12:11:54.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:54 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4293210452' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T12:11:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:54.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:54.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: Reconfiguring daemon mon.a on smithi098 2024-03-16T12:11:54.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:54 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4293210452' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: Reconfiguring daemon mon.a on smithi098 2024-03-16T12:11:54.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:54 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4293210452' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:54.731 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T12:11:54.732 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:11:54.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T12:11:54.815 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:11:54.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:54.953 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:11:54.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T12:11:54.989 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:11:54.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:55.060 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:11:55.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T12:11:55.091 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:11:55.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:55.158 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-03-16T12:11:55.158 INFO:tasks.cephadm:Adding mgr.y on smithi153 2024-03-16T12:11:55.158 INFO:tasks.cephadm:Adding mgr.z on smithi173 2024-03-16T12:11:55.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi098=x;smithi153=y;smithi173=z' 2024-03-16T12:11:55.198 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:55 smithi098 ceph-mon[31633]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:55.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:55 smithi153 ceph-mon[36901]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:55.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:55 smithi173 ceph-mon[36632]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:55.427 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:11:56.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 -- 172.21.15.173:0/1130071202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 msgr2=0x7fba041038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:56.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 --2- 172.21.15.173:0/1130071202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba041038a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009910 tx=0x7fb9f8009c20 comp rx=0 tx=0).stop 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 -- 172.21.15.173:0/1130071202 shutdown_connections 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 --2- 172.21.15.173:0/1130071202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba041038a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 -- 172.21.15.173:0/1130071202 >> 172.21.15.173:0/1130071202 conn(0x7fba040fd3c0 msgr2=0x7fba040ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 -- 172.21.15.173:0/1130071202 shutdown_connections 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.781+0000 7fba09256700 1 -- 172.21.15.173:0/1130071202 wait complete. 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.782+0000 7fba09256700 1 Processor -- start 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.782+0000 7fba09256700 1 -- start start 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.782+0000 7fba09256700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba04076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba040746c0 0x7fba04074b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba04107ef0 con 0x7fba04103480 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba04107d70 con 0x7fba040746c0 2024-03-16T12:11:56.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba04108070 con 0x7fba04075080 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51734/0 (socket says 172.21.15.173:51734) 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 -- 172.21.15.173:0/3097992897 learned_addr learned my addr 172.21.15.173:0/3097992897 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0259c700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba040746c0 0x7fba04074b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba02d9d700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba04076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 -- 172.21.15.173:0/3097992897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba040746c0 msgr2=0x7fba04074b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba040746c0 0x7fba04074b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 -- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 msgr2=0x7fba04076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba04076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 -- 172.21.15.173:0/3097992897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba041a1d30 con 0x7fba04075080 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba02d9d700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba04076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba0359e700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00ba50 tx=0x7fb9fc00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:56.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc019070 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc014440 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0075b0 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba041a1fc0 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.783+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba041a2500 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.784+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb9fc01c070 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.784+0000 7fb9ebfff700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 0x7fb9ec03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.784+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba04063900 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.785+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb9fc04b830 con 0x7fba04075080 2024-03-16T12:11:56.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.785+0000 7fba02d9d700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 0x7fb9ec03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:11:56.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.786+0000 7fba02d9d700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 0x7fb9ec03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005330 tx=0x7fb9f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:11:56.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.790+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9fc012500 con 0x7fba04075080 2024-03-16T12:11:56.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:56.997+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi153=y;smithi173=z", "target": ["mon-mgr", ""]}) -- 0x7fba040627a0 con 0x7fb9ec039450 2024-03-16T12:11:56.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.010+0000 7fb9ebfff700 1 -- 172.21.15.173:0/3097992897 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fba040627a0 con 0x7fb9ec039450 2024-03-16T12:11:56.997 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 msgr2=0x7fb9ec03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 0x7fb9ec03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005330 tx=0x7fb9f8005fd0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 msgr2=0x7fba041a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00ba50 tx=0x7fb9fc00bd60 comp rx=0 tx=0).stop 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 shutdown_connections 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba04075080 0x7fba041a1770 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba04103480 0x7fba04076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb9ec039450 0x7fb9ec03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 --2- 172.21.15.173:0/3097992897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba040746c0 0x7fba04074b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:11:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 >> 172.21.15.173:0/3097992897 conn(0x7fba040fd3c0 msgr2=0x7fba040fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:11:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 shutdown_connections 2024-03-16T12:11:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:11:57.012+0000 7fba09256700 1 -- 172.21.15.173:0/3097992897 wait complete. 2024-03-16T12:11:57.251 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.251 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: Reconfiguring mon.b (monmap changed)... 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: Reconfiguring daemon mon.b on smithi153 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:57.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: Reconfiguring mon.b (monmap changed)... 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:57.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: Reconfiguring daemon mon.b on smithi153 2024-03-16T12:11:57.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:57.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: Reconfiguring mon.b (monmap changed)... 2024-03-16T12:11:57.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:57.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: Reconfiguring daemon mon.b on smithi153 2024-03-16T12:11:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:57.537 DEBUG:teuthology.orchestra.run.smithi153:mgr.y> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T12:11:57.539 DEBUG:teuthology.orchestra.run.smithi173:mgr.z> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T12:11:57.542 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T12:11:57.542 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:11:57.542 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T12:11:57.602 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T12:11:57.602 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 12:10:59.438711871 +0000 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 12:05:52.244827589 +0000 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 12:05:52.244827589 +0000 2024-03-16T12:11:57.653 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 12:05:52.244827589 +0000 2024-03-16T12:11:57.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T12:11:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T12:11:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T12:11:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000517743 s, 989 kB/s 2024-03-16T12:11:57.740 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T12:11:57.795 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-16T12:11:57.843 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 12:10:59.528710694 +0000 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 12:05:52.861819432 +0000 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 12:05:52.861819432 +0000 2024-03-16T12:11:57.844 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 12:05:52.861819432 +0000 2024-03-16T12:11:57.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T12:11:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T12:11:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T12:11:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000359376 s, 1.4 MB/s 2024-03-16T12:11:57.935 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T12:11:57.991 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-16T12:11:58.040 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 12:10:59.613709583 +0000 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 12:05:53.413812135 +0000 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 12:05:53.413812135 +0000 2024-03-16T12:11:58.041 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 12:05:53.413812135 +0000 2024-03-16T12:11:58.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T12:11:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T12:11:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T12:11:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000381303 s, 1.3 MB/s 2024-03-16T12:11:58.131 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T12:11:58.186 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 12:10:59.681708694 +0000 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 12:05:53.967804812 +0000 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 12:05:53.967804812 +0000 2024-03-16T12:11:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 12:05:53.967804812 +0000 2024-03-16T12:11:58.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T12:11:58.325 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-16T12:11:58.325 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-16T12:11:58.325 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000426778 s, 1.2 MB/s 2024-03-16T12:11:58.329 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T12:11:58.381 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-16T12:11:58.382 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T12:11:58.400 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T12:11:58.400 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-16T12:11:58.458 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T12:11:58.458 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 12:11:44.705868212 +0000 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 12:04:36.820178387 +0000 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 12:04:36.820178387 +0000 2024-03-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 12:04:36.820178387 +0000 2024-03-16T12:11:58.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T12:11:58.528 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T12:11:58.529 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T12:11:58.529 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000238365 s, 2.1 MB/s 2024-03-16T12:11:58.530 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T12:11:58.588 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 12:11:44.706868195 +0000 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 12:04:37.167172308 +0000 2024-03-16T12:11:58.645 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 12:04:37.167172308 +0000 2024-03-16T12:11:58.646 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 12:04:37.167172308 +0000 2024-03-16T12:11:58.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T12:11:58.713 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T12:11:58.714 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T12:11:58.714 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000237166 s, 2.2 MB/s 2024-03-16T12:11:58.715 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T12:11:58.772 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi153=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: Saving service mgr spec with placement smithi098=x;smithi153=y;smithi173=z;count:3 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: Reconfiguring mon.c (monmap changed)... 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:58.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:58.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:58 smithi173 ceph-mon[36632]: Reconfiguring daemon mon.c on smithi173 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 12:11:44.706868195 +0000 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 12:04:37.491166631 +0000 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 12:04:37.491166631 +0000 2024-03-16T12:11:58.828 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 12:04:37.491166631 +0000 2024-03-16T12:11:58.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi153=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: Saving service mgr spec with placement smithi098=x;smithi153=y;smithi173=z;count:3 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: Reconfiguring mon.c (monmap changed)... 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:58.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:58.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:58.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:58 smithi153 ceph-mon[36901]: Reconfiguring daemon mon.c on smithi173 2024-03-16T12:11:58.895 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T12:11:58.895 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T12:11:58.896 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000204396 s, 2.5 MB/s 2024-03-16T12:11:58.897 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T12:11:58.952 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-16T12:11:58.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi153=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: Saving service mgr spec with placement smithi098=x;smithi153=y;smithi173=z;count:3 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: Reconfiguring mon.c (monmap changed)... 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:58 smithi098 ceph-mon[31633]: Reconfiguring daemon mon.c on smithi173 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-16 12:11:44.707868178 +0000 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-16 12:04:37.814160973 +0000 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-16 12:04:37.814160973 +0000 2024-03-16T12:11:59.008 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-16 12:04:37.814160973 +0000 2024-03-16T12:11:59.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T12:11:59.075 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-16T12:11:59.075 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-16T12:11:59.075 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000247181 s, 2.1 MB/s 2024-03-16T12:11:59.076 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T12:11:59.132 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-16T12:11:59.133 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T12:11:59.147 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T12:11:59.148 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-16T12:11:59.203 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T12:11:59.203 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-16 12:11:35.441795754 +0000 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-16 12:04:38.570514907 +0000 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-16 12:04:38.570514907 +0000 2024-03-16T12:11:59.204 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-16 12:04:38.570514907 +0000 2024-03-16T12:11:59.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T12:11:59.270 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-16T12:11:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-16T12:11:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000276904 s, 1.8 MB/s 2024-03-16T12:11:59.272 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T12:11:59.327 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-16T12:11:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-16 12:11:35.442795731 +0000 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-16 12:04:38.893507717 +0000 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-16 12:04:38.893507717 +0000 2024-03-16T12:11:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-16 12:04:38.893507717 +0000 2024-03-16T12:11:59.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T12:11:59.448 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-16T12:11:59.448 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-16T12:11:59.448 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00018411 s, 2.8 MB/s 2024-03-16T12:11:59.449 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T12:11:59.505 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-16 12:11:35.442795731 +0000 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-16 12:04:39.233500150 +0000 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-16 12:04:39.233500150 +0000 2024-03-16T12:11:59.562 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-16 12:04:39.233500150 +0000 2024-03-16T12:11:59.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T12:11:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-16T12:11:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-16T12:11:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000191321 s, 2.7 MB/s 2024-03-16T12:11:59.630 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T12:11:59.686 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-16 12:11:35.442795731 +0000 2024-03-16T12:11:59.740 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-16 12:04:39.572492604 +0000 2024-03-16T12:11:59.741 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-16 12:04:39.572492604 +0000 2024-03-16T12:11:59.741 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-16 12:04:39.572492604 +0000 2024-03-16T12:11:59.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T12:11:59.806 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-16T12:11:59.807 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-16T12:11:59.807 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000200996 s, 2.5 MB/s 2024-03-16T12:11:59.808 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T12:11:59.863 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-16T12:11:59.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:11:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:11:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:59.945 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:11:59.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:11:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:00.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:00.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:00.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:12:00.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:00.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:11:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:12:00.739 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:00 smithi098 ceph-mon[31633]: Deploying daemon mgr.z on smithi173 2024-03-16T12:12:00.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:00 smithi173 ceph-mon[36632]: Deploying daemon mgr.z on smithi173 2024-03-16T12:12:01.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:00 smithi153 ceph-mon[36901]: Deploying daemon mgr.z on smithi173 2024-03-16T12:12:01.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:01 smithi173 ceph-mon[36632]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:01.913 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:01 smithi098 ceph-mon[31633]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:02.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:01 smithi153 ceph-mon[36901]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:02.325 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:02.167+0000 7fed94b1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T12:12:02.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:02.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:02.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:02.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:02.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:12:02.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:02.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:02.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:02 smithi098 ceph-mon[31633]: Deploying daemon mgr.y on smithi153 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:03.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:02 smithi153 ceph-mon[36901]: Deploying daemon mgr.y on smithi153 2024-03-16T12:12:03.012 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:02.798+0000 7fed94b1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T12:12:03.012 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:02.887+0000 7fed94b1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T12:12:03.012 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.025+0000 7fed94b1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:03.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:02 smithi173 ceph-mon[36632]: Deploying daemon mgr.y on smithi153 2024-03-16T12:12:03.059 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T12:12:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T12:12:03.060 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T12:12:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T12:12:03.060 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0533235 s, 197 MB/s 2024-03-16T12:12:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T12:12:03.325 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.118+0000 7fed94b1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T12:12:03.325 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.198+0000 7fed94b1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T12:12:03.722 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.458+0000 7fed94b1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T12:12:03.722 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.564+0000 7fed94b1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T12:12:03.722 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.647+0000 7fed94b1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T12:12:03.722 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:03 smithi153 systemd[1]: Starting Ceph mgr.y for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:12:03.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-16T12:12:03.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:03 smithi153 ceph-mon[36901]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:03.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:03 smithi153 podman[38267]: 2024-03-16 12:12:03.767744929 +0000 UTC m=+0.018149789 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:12:03.981 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:03 smithi153 podman[38267]: 2024-03-16 12:12:03.867124504 +0000 UTC m=+0.117529350 container create 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:12:03.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:03 smithi098 ceph-mon[31633]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:04.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-mon[36632]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:04.075 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.742+0000 7fed94b1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T12:12:04.075 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:03.916+0000 7fed94b1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T12:12:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:12:04.425 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 podman[38267]: 2024-03-16 12:12:04.069457603 +0000 UTC m=+0.319862472 container init 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T12:12:04.425 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 podman[38267]: 2024-03-16 12:12:04.072015376 +0000 UTC m=+0.322420228 container start 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T12:12:04.425 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 ceph-mgr[38286]: -- 172.21.15.153:0/238190769 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562feb83a5a0 con 0x562feb80f000 2024-03-16T12:12:04.425 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 bash[38267]: 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a 2024-03-16T12:12:04.425 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 systemd[1]: Started Ceph mgr.y for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:12:04.745 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:04.660+0000 7f39d7642200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T12:12:05.235 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:04.890+0000 7fed94b1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T12:12:05.236 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:05.027+0000 7fed94b1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T12:12:05.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:05.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:05 smithi098 ceph-mon[31633]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:05.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-mon[36901]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:05.508 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:04 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:04.759+0000 7f39d7642200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T12:12:05.508 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:05.466+0000 7f39d7642200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T12:12:05.549 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-mon[36632]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:05.378+0000 7fed94b1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T12:12:05.550 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:05.472+0000 7fed94b1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T12:12:05.759 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:05.564+0000 7f39d7642200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T12:12:05.759 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:05.712+0000 7f39d7642200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T12:12:05.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:05.668+0000 7fed94b1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:12:06.120 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:05.857+0000 7fed94b1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T12:12:06.179 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:05.808+0000 7f39d7642200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T12:12:06.179 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:05.890+0000 7f39d7642200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T12:12:06.179 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:06.192+0000 7f39d7642200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 -- 172.21.15.98:0/1409734297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60102320 msgr2=0x7f8b60102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 --2- 172.21.15.98:0/1409734297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60102320 0x7f8b60102740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8b500099c0 tx=0x7f8b50009cd0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 -- 172.21.15.98:0/1409734297 shutdown_connections 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 --2- 172.21.15.98:0/1409734297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60102320 0x7f8b60102740 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 --2- 172.21.15.98:0/1409734297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60103ec0 0x7f8b6010a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 --2- 172.21.15.98:0/1409734297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103520 0x7f8b60103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.327+0000 7f8b65a00700 1 -- 172.21.15.98:0/1409734297 >> 172.21.15.98:0/1409734297 conn(0x7f8b600fd880 msgr2=0x7f8b600ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:12:06.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.328+0000 7f8b65a00700 1 -- 172.21.15.98:0/1409734297 shutdown_connections 2024-03-16T12:12:06.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.328+0000 7f8b65a00700 1 -- 172.21.15.98:0/1409734297 wait complete. 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.329+0000 7f8b65a00700 1 Processor -- start 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.329+0000 7f8b65a00700 1 -- start start 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.329+0000 7f8b65a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.329+0000 7f8b65a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 0x7f8b6019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b65a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103ec0 0x7f8b601a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b65a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6010df40 con 0x7f8b60102320 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b65a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6010ddc0 con 0x7f8b60103520 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b65a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6010e0c0 con 0x7f8b60103ec0 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 0x7f8b6019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48414/0 (socket says 172.21.15.98:48414) 2024-03-16T12:12:06.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 -- 172.21.15.98:0/248247292 learned_addr learned my addr 172.21.15.98:0/248247292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:12:06.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5f400700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103ec0 0x7f8b601a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 -- 172.21.15.98:0/248247292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103ec0 msgr2=0x7f8b601a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103ec0 0x7f8b601a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 -- 172.21.15.98:0/248247292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 msgr2=0x7f8b6019ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 0x7f8b6019ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5ea00700 1 -- 172.21.15.98:0/248247292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b601a23c0 con 0x7f8b60102320 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.330+0000 7f8b5e000700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 0x7f8b6019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b5ea00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000bee0 tx=0x7f8b5000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50009e60 con 0x7f8b60102320 2024-03-16T12:12:06.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b50003e30 con 0x7f8b60102320 2024-03-16T12:12:06.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b601a2650 con 0x7f8b60102320 2024-03-16T12:12:06.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b500173f0 con 0x7f8b60102320 2024-03-16T12:12:06.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.331+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b601a2b00 con 0x7f8b60102320 2024-03-16T12:12:06.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.332+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f8b500052d0 con 0x7f8b60102320 2024-03-16T12:12:06.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.333+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b60068f90 con 0x7f8b60102320 2024-03-16T12:12:06.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.333+0000 7f8b57400700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 0x7f8b4403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:06.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.333+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8b5004cef0 con 0x7f8b60102320 2024-03-16T12:12:06.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.334+0000 7f8b5e000700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 0x7f8b4403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:06.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.334+0000 7f8b5e000700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 0x7f8b4403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c011490 tx=0x7f8b4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:12:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.339+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b50013070 con 0x7f8b60102320 2024-03-16T12:12:06.382 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:06.134+0000 7fed94b1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:12:06.471 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:06.304+0000 7f39d7642200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T12:12:06.472 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:06.391+0000 7f39d7642200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T12:12:06.472 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:06.485+0000 7f39d7642200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T12:12:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:06.573+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8b60075c20 con 0x7f8b44039200 2024-03-16T12:12:06.759 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:06.659+0000 7f39d7642200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T12:12:06.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:06.396+0000 7fed94b1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T12:12:06.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:06.624+0000 7fed94b1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:07.134 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:07.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:06 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:07.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:07.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:07.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:07.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:06 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:07.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:06 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:07.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:07.403+0000 7fed94b1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T12:12:07.880 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:07.633+0000 7f39d7642200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T12:12:07.881 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:07.763+0000 7f39d7642200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T12:12:08.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:07 smithi098 ceph-mon[31633]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T12:12:08.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:07 smithi098 ceph-mon[31633]: Reconfiguring daemon mgr.x on smithi098 2024-03-16T12:12:08.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:07 smithi098 ceph-mon[31633]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:08.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:07 smithi098 ceph-mon[31633]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:08.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:07 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:08.173 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-mon[36901]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T12:12:08.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-mon[36901]: Reconfiguring daemon mgr.x on smithi098 2024-03-16T12:12:08.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-mon[36901]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:08.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-mon[36901]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:08.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:07 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:08.174 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:08.099+0000 7f39d7642200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T12:12:08.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-mon[36632]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-mon[36632]: Reconfiguring daemon mgr.x on smithi098 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-mon[36632]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-mon[36632]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:07.965+0000 7fed94b1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:08.058+0000 7fed94b1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T12:12:08.253 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z[38495]: 2024-03-16T12:12:08.266+0000 7fed94b1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T12:12:08.509 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:08.187+0000 7f39d7642200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T12:12:08.509 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:08.380+0000 7f39d7642200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:12:08.835 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:08.565+0000 7f39d7642200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T12:12:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:08.906+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f8b50021cc0 con 0x7f8b60102320 2024-03-16T12:12:09.098 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:08.849+0000 7f39d7642200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T12:12:09.098 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:09.111+0000 7f39d7642200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T12:12:09.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T12:12:09.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: Standby manager daemon z started 2024-03-16T12:12:09.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:09.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:08 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T12:12:09.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: Standby manager daemon z started 2024-03-16T12:12:09.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:09.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T12:12:09.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:09.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:08 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: Standby manager daemon z started 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.173:0/3729908686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:09.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:09.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:09.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:08 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:09.509 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:09.345+0000 7f39d7642200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T12:12:09.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:09 smithi098 ceph-mon[31633]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:09.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:09 smithi098 ceph-mon[31633]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-16T12:12:09.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T12:12:10.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:09 smithi153 ceph-mon[36901]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:10.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:09 smithi153 ceph-mon[36901]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-16T12:12:10.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T12:12:10.259 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:10.145+0000 7f39d7642200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T12:12:10.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:09 smithi173 ceph-mon[36632]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:10.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:09 smithi173 ceph-mon[36632]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-16T12:12:10.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T12:12:11.009 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:10.717+0000 7f39d7642200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T12:12:11.009 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:10.813+0000 7f39d7642200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T12:12:11.509 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y[38282]: 2024-03-16T12:12:11.024+0000 7f39d7642200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T12:12:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:11.933+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f8b500163a0 con 0x7f8b60102320 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: Standby manager daemon y started 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]: dispatch 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]': finished 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T12:12:12.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: Standby manager daemon y started 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]: dispatch 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]': finished 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T12:12:12.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T12:12:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: Standby manager daemon y started 2024-03-16T12:12:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='mgr.? 172.21.15.153:0/3823823846' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]: dispatch 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1836580911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "464e1266-b151-4bf9-82b5-398b1aa8d1db"}]': finished 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T12:12:12.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:13.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:12 smithi098 ceph-mon[31633]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-16T12:12:13.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T12:12:13.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:12 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/294154335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:13.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:12 smithi153 ceph-mon[36901]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-16T12:12:13.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T12:12:13.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:12 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/294154335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:13.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:12 smithi173 ceph-mon[36632]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-16T12:12:13.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T12:12:13.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:12 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/294154335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:14.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:13 smithi173 ceph-mon[36632]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:14.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:13 smithi098 ceph-mon[31633]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:14.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:13 smithi153 ceph-mon[36901]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:15.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:14 smithi098 ceph-mon[31633]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:15.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:14 smithi153 ceph-mon[36901]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:14 smithi173 ceph-mon[36632]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:17.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:17 smithi098 ceph-mon[31633]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:18.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:17 smithi153 ceph-mon[36901]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:18.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:17 smithi173 ceph-mon[36632]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:19.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:19 smithi098 ceph-mon[31633]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:20.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:19 smithi153 ceph-mon[36901]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:20.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:19 smithi173 ceph-mon[36632]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T12:12:20.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T12:12:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:21.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T12:12:21.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:21.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:21 smithi098 ceph-mon[31633]: Deploying daemon osd.0 on smithi098 2024-03-16T12:12:21.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:21 smithi098 ceph-mon[31633]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:21 smithi153 ceph-mon[36901]: Deploying daemon osd.0 on smithi098 2024-03-16T12:12:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:21 smithi153 ceph-mon[36901]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:22.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:21 smithi173 ceph-mon[36632]: Deploying daemon osd.0 on smithi098 2024-03-16T12:12:22.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:21 smithi173 ceph-mon[36632]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:23.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:23 smithi098 ceph-mon[31633]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:23 smithi153 ceph-mon[36901]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:24.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:23 smithi173 ceph-mon[36632]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:25.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:25 smithi098 ceph-mon[31633]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:26.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:25 smithi153 ceph-mon[36901]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:25 smithi173 ceph-mon[36632]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:27.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:27 smithi098 ceph-mon[31633]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:28.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:27 smithi153 ceph-mon[36901]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:28.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:27 smithi173 ceph-mon[36632]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:29.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:29.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:29 smithi098 ceph-mon[31633]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:29.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:29.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:29 smithi153 ceph-mon[36901]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:29 smithi173 ceph-mon[36632]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:31.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:31.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:31.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:31.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:31.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:31.981 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:32.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:32.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:32.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:32.950 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-03-16T12:12:32.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.962+0000 7f8b57400700 1 -- 172.21.15.98:0/248247292 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8b60075c20 con 0x7f8b44039200 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 msgr2=0x7f8b4403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 0x7f8b4403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c011490 tx=0x7f8b4c009260 comp rx=0 tx=0).stop 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 msgr2=0x7f8b6019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000bee0 tx=0x7f8b5000bf10 comp rx=0 tx=0).stop 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 shutdown_connections 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b60103ec0 0x7f8b601a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b60102320 0x7f8b6019a800 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b44039200 0x7f8b4403b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 --2- 172.21.15.98:0/248247292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b60103520 0x7f8b6019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 >> 172.21.15.98:0/248247292 conn(0x7f8b600fd880 msgr2=0x7f8b60074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:12:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 shutdown_connections 2024-03-16T12:12:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:12:32.966+0000 7f8b65a00700 1 -- 172.21.15.98:0/248247292 wait complete. 2024-03-16T12:12:32.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:32 smithi098 ceph-mon[31633]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:32.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:32 smithi098 ceph-mon[31633]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:32 smithi153 ceph-mon[36901]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:32 smithi153 ceph-mon[36901]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:33.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:32 smithi173 ceph-mon[36632]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:33.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:32 smithi173 ceph-mon[36632]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T12:12:33.780 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T12:12:33.785 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_4... 2024-03-16T12:12:33.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T12:12:33.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:33.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T12:12:33.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T12:12:33.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:33.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:33.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:33.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:33.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:33.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:34.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:34.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:34.023 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:34.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-mon[31633]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:12:34.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-mon[31633]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T12:12:34.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:34.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:34.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-mon[31633]: from='osd.0 ' entity='osd.0' 2024-03-16T12:12:34.999 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0[46633]: 2024-03-16T12:12:34.636+0000 7fa03f200700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T12:12:34.999 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:12:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0[46633]: 2024-03-16T12:12:34.660+0000 7fa035000700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:12:35.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:34 smithi153 ceph-mon[36901]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:12:35.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:34 smithi153 ceph-mon[36901]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T12:12:35.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:34 smithi153 ceph-mon[36901]: from='osd.0 ' entity='osd.0' 2024-03-16T12:12:35.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:34 smithi173 ceph-mon[36632]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:12:35.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:34 smithi173 ceph-mon[36632]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T12:12:35.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:34 smithi173 ceph-mon[36632]: from='osd.0 ' entity='osd.0' 2024-03-16T12:12:35.762 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:12:35.762 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:12:35.762 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:35.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] boot 2024-03-16T12:12:35.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T12:12:35.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T12:12:35.931 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T12:12:35.932 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T12:12:35.932 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T12:12:35.932 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120206 s, 872 MB/s 2024-03-16T12:12:35.932 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] boot 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T12:12:35.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: osd.0 [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] boot 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T12:12:36.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T12:12:36.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-03-16T12:12:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:12:37.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:37 smithi153 ceph-mon[36901]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:37.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:37 smithi153 ceph-mon[36901]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T12:12:37.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:37 smithi098 ceph-mon[31633]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:37.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:37 smithi098 ceph-mon[31633]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T12:12:38.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:37 smithi173 ceph-mon[36632]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:38.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:37 smithi173 ceph-mon[36632]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T12:12:38.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 -- 172.21.15.153:0/3461943187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c103e90 msgr2=0x7f610c10a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 --2- 172.21.15.153:0/3461943187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c103e90 0x7f610c10a740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f610400b0d0 tx=0x7f610400b3e0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 -- 172.21.15.153:0/3461943187 shutdown_connections 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 --2- 172.21.15.153:0/3461943187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c103e90 0x7f610c10a740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 --2- 172.21.15.153:0/3461943187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1022f0 0x7f610c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 --2- 172.21.15.153:0/3461943187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c1034f0 0x7f610c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 -- 172.21.15.153:0/3461943187 >> 172.21.15.153:0/3461943187 conn(0x7f610c0fd8b0 msgr2=0x7f610c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 -- 172.21.15.153:0/3461943187 shutdown_connections 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.380+0000 7f6114031700 1 -- 172.21.15.153:0/3461943187 wait complete. 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 Processor -- start 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 -- start start 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1034f0 0x7f610c0746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 0x7f610c074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f610c10deb0 con 0x7f610c103e90 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f610c10dd30 con 0x7f610c1022f0 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6114031700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f610c10e030 con 0x7f610c1034f0 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6111dcd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6111dcd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59238/0 (socket says 172.21.15.153:59238) 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f6111dcd700 1 -- 172.21.15.153:0/1932639367 learned_addr learned my addr 172.21.15.153:0/1932639367 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.381+0000 7f61115cc700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1034f0 0x7f610c0746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f61125ce700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 0x7f610c074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 -- 172.21.15.153:0/1932639367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1034f0 msgr2=0x7f610c0746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1034f0 0x7f610c0746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 -- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 msgr2=0x7f610c074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 0x7f610c074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 -- 172.21.15.153:0/1932639367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c0754b0 con 0x7f610c1022f0 2024-03-16T12:12:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f61125ce700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 0x7f610c074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:12:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6111dcd700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f610800c990 tx=0x7f610800cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:12:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108007b00 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c1a2570 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6108007ca0 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108018680 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.382+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1a2a60 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.383+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610c068f90 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.384+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f610800f450 con 0x7f610c1022f0 2024-03-16T12:12:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.384+0000 7f6102ffd700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 0x7f60f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:12:38.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.385+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f61080ccf90 con 0x7f610c1022f0 2024-03-16T12:12:38.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.385+0000 7f61115cc700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 0x7f60f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:12:38.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.386+0000 7f61115cc700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 0x7f60f80a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005950 tx=0x7f60fc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:12:38.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.389+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6108014030 con 0x7f610c1022f0 2024-03-16T12:12:38.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:38.597+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f610c062710 con 0x7f60f80a2800 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi098 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi098 to 17091M 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:39 smithi153 ceph-mon[36901]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi098 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi098 to 17091M 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:39.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:39.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:39.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:39 smithi173 ceph-mon[36632]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi098 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi098 to 17091M 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:39.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:12:39.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:12:39.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:12:39.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:39.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:39 smithi098 ceph-mon[31633]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:41.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:41.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/1711033396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:41.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:41.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]': finished 2024-03-16T12:12:41.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T12:12:41.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:42.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:42.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/1711033396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:42.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:42.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]': finished 2024-03-16T12:12:42.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T12:12:42.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:41 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/1711033396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]: dispatch 2024-03-16T12:12:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb71b246-921e-49df-9e6d-08cd3431f968"}]': finished 2024-03-16T12:12:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T12:12:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:42.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/3884144830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:43.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/3884144830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/3884144830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:12:43.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:43 smithi098 ceph-mon[31633]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:44.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:43 smithi153 ceph-mon[36901]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:43 smithi173 ceph-mon[36632]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:45.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:45 smithi153 ceph-mon[36901]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:45.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:45 smithi098 ceph-mon[31633]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:45 smithi173 ceph-mon[36632]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:47.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:47 smithi098 ceph-mon[31633]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:48.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:47 smithi153 ceph-mon[36901]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:47 smithi173 ceph-mon[36632]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:48.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T12:12:48.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:49.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T12:12:49.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T12:12:49.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:50.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:49 smithi153 ceph-mon[36901]: Deploying daemon osd.1 on smithi153 2024-03-16T12:12:50.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:49 smithi153 ceph-mon[36901]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:49 smithi173 ceph-mon[36632]: Deploying daemon osd.1 on smithi153 2024-03-16T12:12:50.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:49 smithi173 ceph-mon[36632]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:50.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:49 smithi098 ceph-mon[31633]: Deploying daemon osd.1 on smithi153 2024-03-16T12:12:50.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:49 smithi098 ceph-mon[31633]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:52.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:51 smithi153 ceph-mon[36901]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:51 smithi173 ceph-mon[36632]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:52.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:51 smithi098 ceph-mon[31633]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:53 smithi153 ceph-mon[36901]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:54.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:53 smithi098 ceph-mon[31633]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:53 smithi173 ceph-mon[36632]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:54.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:54.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:54.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:55.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:55.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:55.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:55.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:55.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:56.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.088 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.088 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:56.088 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:56.088 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:56.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:56.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:12:56.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:12:56.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:57.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.642+0000 7f6102ffd700 1 -- 172.21.15.153:0/1932639367 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f610c062710 con 0x7f60f80a2800 2024-03-16T12:12:57.629 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-03-16T12:12:57.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 msgr2=0x7f60f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 0x7f60f80a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005950 tx=0x7f60fc00b410 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 msgr2=0x7f610c076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f610800c990 tx=0x7f610800cca0 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 shutdown_connections 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f60f80a2800 0x7f60f80a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f610c1022f0 0x7f610c076070 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f610c103e90 0x7f610c074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 --2- 172.21.15.153:0/1932639367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c1034f0 0x7f610c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 >> 172.21.15.153:0/1932639367 conn(0x7f610c0fd8b0 msgr2=0x7f610c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.644+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 shutdown_connections 2024-03-16T12:12:57.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:12:57.645+0000 7f6114031700 1 -- 172.21.15.153:0/1932639367 wait complete. 2024-03-16T12:12:57.997 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:57.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:57.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:57.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:57.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:57.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:57 smithi153 ceph-mon[36901]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:58.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:58.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:58.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:58.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:58.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:57 smithi173 ceph-mon[36632]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:58.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:58.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:58.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:12:58.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:12:58.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:58.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:57 smithi098 ceph-mon[31633]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T12:12:58.323 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T12:12:58.325 INFO:tasks.cephadm:Deploying osd.2 on smithi173 with /dev/vg_nvme/lv_4... 2024-03-16T12:12:58.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T12:12:58.569 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:12:59.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T12:12:59.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T12:12:59.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:59.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:12:59 smithi153 ceph-mon[36901]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:59.864 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T12:12:59.864 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.864 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T12:12:59.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:59.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:12:59 smithi173 ceph-mon[36632]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:12:59.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T12:12:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: from='osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T12:12:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:12:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:12:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:12:59 smithi098 ceph-mon[31633]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:13:00.456 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111485 s, 941 MB/s 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-03-16T12:13:00.457 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-16T12:13:00.616 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1[41485]: 2024-03-16T12:13:00.314+0000 7f455f65e700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T12:13:00.616 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1[41485]: 2024-03-16T12:13:00.320+0000 7f4557449700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:13:00.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:13:00.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T12:13:00.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:00.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:00.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: from='osd.1 ' entity='osd.1' 2024-03-16T12:13:00.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:00 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: from='osd.1 ' entity='osd.1' 2024-03-16T12:13:01.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:00 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: from='osd.1 ' entity='osd.1' 2024-03-16T12:13:01.067 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:00 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-03-16T12:13:01.333 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:13:01.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] boot 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:01.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] boot 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:01.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: osd.1 [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] boot 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:01.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- 172.21.15.173:0/2059271437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04103ea0 msgr2=0x7ffa0410a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 --2- 172.21.15.173:0/2059271437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04103ea0 0x7ffa0410a750 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00b0a0 tx=0x7ff9fc00b3b0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- 172.21.15.173:0/2059271437 shutdown_connections 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 --2- 172.21.15.173:0/2059271437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04103ea0 0x7ffa0410a750 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 --2- 172.21.15.173:0/2059271437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04102300 0x7ffa04102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 --2- 172.21.15.173:0/2059271437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 0x7ffa04103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- 172.21.15.173:0/2059271437 >> 172.21.15.173:0/2059271437 conn(0x7ffa040fd8e0 msgr2=0x7ffa040ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:13:02.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- 172.21.15.173:0/2059271437 shutdown_connections 2024-03-16T12:13:02.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- 172.21.15.173:0/2059271437 wait complete. 2024-03-16T12:13:02.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 Processor -- start 2024-03-16T12:13:02.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.803+0000 7ffa08d45700 1 -- start start 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 0x7ffa04198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04103ea0 0x7ffa0419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410df80 con 0x7ffa04103ea0 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410de00 con 0x7ffa04103500 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0410e100 con 0x7ffa04102300 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:36234/0 (socket says 172.21.15.173:36234) 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 -- 172.21.15.173:0/6724817 learned_addr learned my addr 172.21.15.173:0/6724817 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa01d9b700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 0x7ffa04198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 -- 172.21.15.173:0/6724817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 msgr2=0x7ffa04198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 0x7ffa04198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 -- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04103ea0 msgr2=0x7ffa0419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04103ea0 0x7ffa0419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 -- 172.21.15.173:0/6724817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa041a01a0 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa0259c700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400b530 tx=0x7ff9f400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4015020 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9f4011d10 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041a0490 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f400f2e0 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.804+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a0a50 con 0x7ffa04102300 2024-03-16T12:13:02.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.805+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa04068f90 con 0x7ffa04102300 2024-03-16T12:13:02.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.806+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7ff9f4018020 con 0x7ffa04102300 2024-03-16T12:13:02.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.807+0000 7ff9f37fe700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 0x7ff9ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:02.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.807+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7ff9f40cc930 con 0x7ffa04102300 2024-03-16T12:13:02.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.807+0000 7ffa01d9b700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 0x7ff9ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:02.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.808+0000 7ffa01d9b700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 0x7ff9ec0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009c60 tx=0x7ff9f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:13:02.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:02.812+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9f401f020 con 0x7ffa04102300 2024-03-16T12:13:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:02 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi153 2024-03-16T12:13:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:02 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-16T12:13:02.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:02 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi153 2024-03-16T12:13:02.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:02 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-16T12:13:02.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:02 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi153 2024-03-16T12:13:02.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:02 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi153 to 17080M 2024-03-16T12:13:03.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:03.015+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ffa04062710 con 0x7ff9ec0a2800 2024-03-16T12:13:03.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:03 smithi173 ceph-mon[36632]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T12:13:03.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:03 smithi173 ceph-mon[36632]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:03.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:03 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:03.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:03 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:03.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:03 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:03.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:03 smithi098 ceph-mon[31633]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T12:13:03.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:03 smithi098 ceph-mon[31633]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:03.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:03 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:03.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:03 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:03.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:03 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:04.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:03 smithi153 ceph-mon[36901]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T12:13:04.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:03 smithi153 ceph-mon[36901]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:04.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:03 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:04.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:03 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:04.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:03 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:04.784 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:04 smithi173 ceph-mon[36632]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:04.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:04 smithi098 ceph-mon[31633]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:05.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:04 smithi153 ceph-mon[36901]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/937534560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]': finished 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T12:13:05.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/937534560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]': finished 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T12:13:05.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:06.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:06.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/937534560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:06.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]: dispatch 2024-03-16T12:13:06.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc"}]': finished 2024-03-16T12:13:06.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T12:13:06.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:06.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:06 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/1680123213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:07.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:06 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/1680123213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:06 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/1680123213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:07.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:07 smithi098 ceph-mon[31633]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:08.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:07 smithi153 ceph-mon[36901]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:07 smithi173 ceph-mon[36632]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:09.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:09 smithi173 ceph-mon[36632]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:09.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:09 smithi098 ceph-mon[31633]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:10.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:09 smithi153 ceph-mon[36901]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:11.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:11 smithi098 ceph-mon[31633]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:12.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:11 smithi153 ceph-mon[36901]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:11 smithi173 ceph-mon[36632]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:13.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:13 smithi153 ceph-mon[36901]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:13.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T12:13:13.948 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:13.948 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:13 smithi153 ceph-mon[36901]: Deploying daemon osd.2 on smithi173 2024-03-16T12:13:13.954 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:13 smithi173 ceph-mon[36632]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:13.954 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T12:13:13.954 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:13.954 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:13 smithi173 ceph-mon[36632]: Deploying daemon osd.2 on smithi173 2024-03-16T12:13:13.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:13 smithi098 ceph-mon[31633]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:13.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T12:13:13.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:13.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:13 smithi098 ceph-mon[31633]: Deploying daemon osd.2 on smithi173 2024-03-16T12:13:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:15 smithi173 ceph-mon[36632]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:15.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:15 smithi098 ceph-mon[31633]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:16.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:15 smithi153 ceph-mon[36901]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:17.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:17 smithi098 ceph-mon[31633]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:18.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:17 smithi153 ceph-mon[36901]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:17 smithi173 ceph-mon[36632]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:19.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:19.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:18 smithi098 ceph-mon[31633]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:19.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:19.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:18 smithi153 ceph-mon[36901]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:18 smithi173 ceph-mon[36632]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:21.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.245+0000 7ff9f37fe700 1 -- 172.21.15.173:0/6724817 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffa04062710 con 0x7ff9ec0a2800 2024-03-16T12:13:21.232 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 2 on host 'smithi173' 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 msgr2=0x7ff9ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 0x7ff9ec0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009c60 tx=0x7ff9f8009260 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 msgr2=0x7ffa04198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400b530 tx=0x7ff9f400b840 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 shutdown_connections 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa04103ea0 0x7ffa0419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa04102300 0x7ffa04198540 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff9ec0a2800 0x7ff9ec0a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 --2- 172.21.15.173:0/6724817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa04103500 0x7ffa04198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:21.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 >> 172.21.15.173:0/6724817 conn(0x7ffa040fd8e0 msgr2=0x7ffa041067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:13:21.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 shutdown_connections 2024-03-16T12:13:21.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:13:21.249+0000 7ffa08d45700 1 -- 172.21.15.173:0/6724817 wait complete. 2024-03-16T12:13:21.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:21 smithi173 ceph-mon[36632]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:21.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.901 DEBUG:teuthology.orchestra.run.smithi173:osd.2> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T12:13:21.903 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-16T12:13:21.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T12:13:21.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:21 smithi098 ceph-mon[31633]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:21.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:21.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:21 smithi153 ceph-mon[36901]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:13:22.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:22 smithi098 ceph-mon[31633]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:22.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:22 smithi098 ceph-mon[31633]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:23.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:22 smithi173 ceph-mon[36632]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:23.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:22 smithi173 ceph-mon[36632]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:23.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:22 smithi153 ceph-mon[36901]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:23.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:22 smithi153 ceph-mon[36901]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T12:13:23.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:23.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T12:13:23.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:23.889 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T12:13:23.889 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:23.889 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:23 smithi098 ceph-mon[31633]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:24.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:23 smithi153 ceph-mon[36901]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: from='osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:23 smithi173 ceph-mon[36632]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:13:24.575 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2[41753]: 2024-03-16T12:13:24.493+0000 7f7a23411700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T12:13:24.575 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2[41753]: 2024-03-16T12:13:24.498+0000 7f7a1a9fb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:13:24.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-mon[36632]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:13:24.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-mon[36632]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T12:13:24.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:24.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-mon[36632]: from='osd.2 ' entity='osd.2' 2024-03-16T12:13:24.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:25.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:24 smithi153 ceph-mon[36901]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:13:25.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:24 smithi153 ceph-mon[36901]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T12:13:25.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:25.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:24 smithi153 ceph-mon[36901]: from='osd.2 ' entity='osd.2' 2024-03-16T12:13:25.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:25.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:24 smithi098 ceph-mon[31633]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:13:25.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:24 smithi098 ceph-mon[31633]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T12:13:25.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:25.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:24 smithi098 ceph-mon[31633]: from='osd.2 ' entity='osd.2' 2024-03-16T12:13:25.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:25.113 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T12:13:25.113 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T12:13:25.113 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T12:13:25.113 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T12:13:25.114 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0373755 s, 281 MB/s 2024-03-16T12:13:25.114 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T12:13:25.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] boot 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T12:13:26.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] boot 2024-03-16T12:13:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T12:13:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:26.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:13:26.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:13:26.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T12:13:26.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: osd.2 [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] boot 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T12:13:26.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T12:13:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:13:27.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:27 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi173 2024-03-16T12:13:27.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:27 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-16T12:13:27.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:27 smithi098 ceph-mon[31633]: pgmap v90: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:27.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:27 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi173 2024-03-16T12:13:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:27 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-16T12:13:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:27 smithi173 ceph-mon[36632]: pgmap v90: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:27.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:27 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi173 2024-03-16T12:13:27.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:27 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi173 to 17080M 2024-03-16T12:13:27.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:27 smithi153 ceph-mon[36901]: pgmap v90: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:27.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.467+0000 7f856d000700 1 -- 172.21.15.98:0/2386230393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 msgr2=0x7f8568108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.467+0000 7f856d000700 1 --2- 172.21.15.98:0/2386230393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f8568108c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8558009920 tx=0x7f8558009c30 comp rx=0 tx=0).stop 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 -- 172.21.15.98:0/2386230393 shutdown_connections 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 --2- 172.21.15.98:0/2386230393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f8568108c80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 --2- 172.21.15.98:0/2386230393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 0x7f8568110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 --2- 172.21.15.98:0/2386230393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f8568109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:28.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 -- 172.21.15.98:0/2386230393 >> 172.21.15.98:0/2386230393 conn(0x7f85680777e0 msgr2=0x7f8568079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:13:28.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 -- 172.21.15.98:0/2386230393 shutdown_connections 2024-03-16T12:13:28.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.468+0000 7f856d000700 1 -- 172.21.15.98:0/2386230393 wait complete. 2024-03-16T12:13:28.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.469+0000 7f856d000700 1 Processor -- start 2024-03-16T12:13:28.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.469+0000 7f856d000700 1 -- start start 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.469+0000 7f856d000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f856d000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f856d000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 0x7f85681a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f856d000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8568114420 con 0x7f856810a3b0 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f856d000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85681142a0 con 0x7f8568108860 2024-03-16T12:13:28.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f856d000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85681145a0 con 0x7f8568109a10 2024-03-16T12:13:28.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:28.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8566a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:28.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:41210/0 (socket says 172.21.15.98:41210) 2024-03-16T12:13:28.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 0x7f85681a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:28.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8566a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52702/0 (socket says 172.21.15.98:52702) 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 -- 172.21.15.98:0/908675757 learned_addr learned my addr 172.21.15.98:0/908675757 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 -- 172.21.15.98:0/908675757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 msgr2=0x7f856819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 -- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 msgr2=0x7f85681a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 0x7f85681a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8567400700 1 -- 172.21.15.98:0/908675757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85681a66d0 con 0x7f8568108860 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.470+0000 7f8566a00700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f8567400700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8558005250 tx=0x7f8558005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558027070 con 0x7f8568108860 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f855800f930 con 0x7f8568108860 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558004af0 con 0x7f8568108860 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85681a6960 con 0x7f8568108860 2024-03-16T12:13:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.471+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85681a6d60 con 0x7f8568108860 2024-03-16T12:13:28.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.472+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8568068f90 con 0x7f8568108860 2024-03-16T12:13:28.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.474+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f855802a070 con 0x7f8568108860 2024-03-16T12:13:28.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.475+0000 7f855fe00700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 0x7f854c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:13:28.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.475+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f85580cdfe0 con 0x7f8568108860 2024-03-16T12:13:28.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.476+0000 7f8566a00700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 0x7f854c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:13:28.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.476+0000 7f8566a00700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 0x7f854c0a4d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8554011490 tx=0x7f8554009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:13:28.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.480+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f855805e740 con 0x7f8568108860 2024-03-16T12:13:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:28.725+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8568062710 con 0x7f854c0a2880 2024-03-16T12:13:28.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:28.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:28 smithi098 ceph-mon[31633]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T12:13:28.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:28.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:28.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:28 smithi153 ceph-mon[36901]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T12:13:28.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:28 smithi173 ceph-mon[36632]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T12:13:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:13:29.590 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:29.590 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:29.590 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:29.590 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51142]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:29.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:29.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T12:13:29.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:29.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:29.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:29.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:29.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:29.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:29.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:13:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T12:13:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:13:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:13:29.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:13:29.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:30.247 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51190]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:30.247 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51190]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:30.247 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51190]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:30.247 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:13:29 smithi098 sudo[51190]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:30.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:30 smithi153 ceph-mon[36901]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T12:13:30.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:30.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45517]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:30.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:30.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45517]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:30.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45517]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:30 smithi098 ceph-mon[31633]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T12:13:30.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:30.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:30 smithi173 ceph-mon[36632]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T12:13:30.751 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:13:30 smithi153 sudo[45520]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:31.004 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:30 smithi173 sudo[45404]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:31.004 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:30 smithi173 sudo[45404]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:31.004 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:30 smithi173 sudo[45404]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:31.004 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:30 smithi173 sudo[45404]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:31.324 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:31 smithi173 sudo[45407]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:31.324 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:31 smithi173 sudo[45407]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:31.324 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:31 smithi173 sudo[45407]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:31.324 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:13:31 smithi173 sudo[45407]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:31.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:31 smithi098 sudo[51238]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:31.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:31 smithi098 sudo[51238]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:31.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:31 smithi098 sudo[51238]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:31.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:31 smithi098 sudo[51238]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:32.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:32.180+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f85580984d0 con 0x7f8568108860 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 sudo[51260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 sudo[51260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 sudo[51260]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 sudo[51260]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 ceph-mon[31633]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:32.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:32 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:32.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:32.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 ceph-mon[36901]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:32.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:32.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45410]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:32.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45410]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:32.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45410]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:32.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45410]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45413]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 sudo[45413]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:33.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:33.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 ceph-mon[36632]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:33.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:32 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:33.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 sudo[45523]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T12:13:33.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 sudo[45523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:33.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 sudo[45523]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:33.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:32 smithi153 sudo[45523]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T12:13:33.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3686449558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]': finished 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:33.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:33 smithi098 ceph-mon[31633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 sudo[45526]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 sudo[45526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 sudo[45526]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 sudo[45526]: pam_unix(sudo:session): session closed for user root 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3686449558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]': finished 2024-03-16T12:13:33.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:33.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T12:13:33.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:33.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:33 smithi153 ceph-mon[36901]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3686449558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T12:13:33.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T12:13:33.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e1de00e-5fc6-4a53-9cfa-99cede3449aa"}]': finished 2024-03-16T12:13:33.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:33.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T12:13:33.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:33.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:33 smithi173 ceph-mon[36632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T12:13:34.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:34 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3813659390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:35.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:34 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3813659390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:34 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3813659390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:13:35.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:35 smithi098 ceph-mon[31633]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:36.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:35 smithi153 ceph-mon[36901]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:35 smithi173 ceph-mon[36632]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:37.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:37 smithi098 ceph-mon[31633]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-16T12:13:38.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:37 smithi153 ceph-mon[36901]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-16T12:13:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:37 smithi173 ceph-mon[36632]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-16T12:13:39.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:39 smithi098 ceph-mon[31633]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-16T12:13:40.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:39 smithi153 ceph-mon[36901]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-16T12:13:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:39 smithi173 ceph-mon[36632]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-16T12:13:41.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:41 smithi098 ceph-mon[31633]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-16T12:13:42.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:41 smithi153 ceph-mon[36901]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-16T12:13:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:41 smithi173 ceph-mon[36632]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-16T12:13:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T12:13:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:43.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:42 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T12:13:43.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:42 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T12:13:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:43.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:43 smithi098 ceph-mon[31633]: Deploying daemon osd.3 on smithi098 2024-03-16T12:13:43.950 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:43 smithi098 ceph-mon[31633]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-16T12:13:44.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:43 smithi153 ceph-mon[36901]: Deploying daemon osd.3 on smithi098 2024-03-16T12:13:44.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:43 smithi153 ceph-mon[36901]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-16T12:13:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:43 smithi173 ceph-mon[36632]: Deploying daemon osd.3 on smithi098 2024-03-16T12:13:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:43 smithi173 ceph-mon[36632]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-16T12:13:45.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:45 smithi098 ceph-mon[31633]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:45 smithi153 ceph-mon[36901]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:45 smithi173 ceph-mon[36632]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:47.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:47 smithi098 ceph-mon[31633]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:48.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:47 smithi153 ceph-mon[36901]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:47 smithi173 ceph-mon[36632]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-16T12:13:49.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:49 smithi098 ceph-mon[31633]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:50.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:49 smithi153 ceph-mon[36901]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:49 smithi173 ceph-mon[36632]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:51.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:51.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:51 smithi098 ceph-mon[31633]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:51 smithi173 ceph-mon[36632]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:51.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:51.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:51.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:51 smithi153 ceph-mon[36901]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:53.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:54.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:13:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:54.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:54 smithi098 ceph-mon[31633]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T12:13:55.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:54 smithi153 ceph-mon[36901]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T12:13:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:54 smithi173 ceph-mon[36632]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T12:13:55.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.371+0000 7f855fe00700 1 -- 172.21.15.98:0/908675757 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8568062710 con 0x7f854c0a2880 2024-03-16T12:13:55.357 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-03-16T12:13:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.374+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 msgr2=0x7f854c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.374+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 0x7f854c0a4d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8554011490 tx=0x7f8554009260 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.374+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 msgr2=0x7f856819ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.374+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8558005250 tx=0x7f8558005280 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 shutdown_connections 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f854c0a2880 0x7f854c0a4d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8568108860 0x7f856819ea70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f856810a3b0 0x7f85681a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 --2- 172.21.15.98:0/908675757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8568109a10 0x7f856819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 >> 172.21.15.98:0/908675757 conn(0x7f85680777e0 msgr2=0x7f856810cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 shutdown_connections 2024-03-16T12:13:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:13:55.375+0000 7f856d000700 1 -- 172.21.15.98:0/908675757 wait complete. 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:55.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:56.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:13:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:13:56.120 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T12:13:56.124 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_3... 2024-03-16T12:13:56.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T12:13:56.371 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:13:57.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:56 smithi153 ceph-mon[36901]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:13:57.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:56 smithi153 ceph-mon[36901]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T12:13:57.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:56 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:56 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:56 smithi153 ceph-mon[36901]: osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] boot 2024-03-16T12:13:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:56 smithi173 ceph-mon[36632]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:13:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:56 smithi173 ceph-mon[36632]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T12:13:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:56 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:56 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:56 smithi173 ceph-mon[36632]: osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] boot 2024-03-16T12:13:57.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:56 smithi098 ceph-mon[31633]: from='osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:13:57.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:56 smithi098 ceph-mon[31633]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T12:13:57.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:56 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:56 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:56 smithi098 ceph-mon[31633]: osd.3 [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] boot 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:57 smithi153 ceph-mon[36901]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:57.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:57 smithi098 ceph-mon[31633]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T12:13:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:13:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:13:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-16T12:13:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T12:13:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T12:13:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:57 smithi173 ceph-mon[36632]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114806 s, 913 MB/s 2024-03-16T12:13:58.443 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T12:13:59.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-03-16T12:13:59.323 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:13:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:59 smithi153 ceph-mon[36901]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T12:13:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:13:59 smithi153 ceph-mon[36901]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:13:59.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:59 smithi098 ceph-mon[31633]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T12:13:59.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:13:59 smithi098 ceph-mon[31633]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:13:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:59 smithi173 ceph-mon[36632]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T12:13:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:13:59 smithi173 ceph-mon[36632]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:00.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:00 smithi153 ceph-mon[36901]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T12:14:00.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:00 smithi098 ceph-mon[31633]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T12:14:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:00 smithi173 ceph-mon[36632]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T12:14:00.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:00.999+0000 7fbf6384b700 1 -- 172.21.15.153:0/254744558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1022f0 msgr2=0x7fbf5c102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:00.999+0000 7fbf6384b700 1 --2- 172.21.15.153:0/254744558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c102710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0099c0 tx=0x7fbf4c009cd0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 -- 172.21.15.153:0/254744558 shutdown_connections 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 --2- 172.21.15.153:0/254744558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c102710 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 --2- 172.21.15.153:0/254744558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c103e90 0x7fbf5c10a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 --2- 172.21.15.153:0/254744558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 0x7fbf5c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 -- 172.21.15.153:0/254744558 >> 172.21.15.153:0/254744558 conn(0x7fbf5c0fd8b0 msgr2=0x7fbf5c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 -- 172.21.15.153:0/254744558 shutdown_connections 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 -- 172.21.15.153:0/254744558 wait complete. 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 Processor -- start 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 -- start start 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.000+0000 7fbf6384b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 0x7fbf5c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c10dfb0 con 0x7fbf5c1034f0 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c10de30 con 0x7fbf5c103e90 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c10e130 con 0x7fbf5c1022f0 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53674/0 (socket says 172.21.15.153:53674) 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 -- 172.21.15.153:0/2821386964 learned_addr learned my addr 172.21.15.153:0/2821386964 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf615e7700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf60de6700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 0x7fbf5c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 -- 172.21.15.153:0/2821386964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c1022f0 msgr2=0x7fbf5c19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 -- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 msgr2=0x7fbf5c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 0x7fbf5c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 -- 172.21.15.153:0/2821386964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5c1a24b0 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf61de8700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400b970 tx=0x7fbf5400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54019070 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5c1a27a0 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf5400ce10 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540073d0 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.001+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf5c1a2d10 con 0x7fbf5c103e90 2024-03-16T12:14:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.002+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf5c068f90 con 0x7fbf5c103e90 2024-03-16T12:14:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.003+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbf5401c070 con 0x7fbf5c103e90 2024-03-16T12:14:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.004+0000 7fbf527fc700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 0x7fbf480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.004+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fbf540cd090 con 0x7fbf5c103e90 2024-03-16T12:14:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.004+0000 7fbf615e7700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 0x7fbf480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.005+0000 7fbf615e7700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 0x7fbf480a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00b580 tx=0x7fbf4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.008+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf5409b350 con 0x7fbf5c103e90 2024-03-16T12:14:01.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:01.209+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbf5c10a3b0 con 0x7fbf480a2890 2024-03-16T12:14:01.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:01 smithi153 ceph-mon[36901]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T12:14:01.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:01 smithi153 ceph-mon[36901]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:01.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:01.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:01.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:01.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:01 smithi098 ceph-mon[31633]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T12:14:01.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:01 smithi098 ceph-mon[31633]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:01.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:01.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:01.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:01 smithi173 ceph-mon[36632]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T12:14:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:01 smithi173 ceph-mon[36632]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:02.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:02.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:02.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:03.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:03 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi098 2024-03-16T12:14:03.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:03 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi098 to 8545M 2024-03-16T12:14:03.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:03 smithi098 ceph-mon[31633]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-16T12:14:03.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:03 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi098 2024-03-16T12:14:03.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:03 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi098 to 8545M 2024-03-16T12:14:03.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:03 smithi153 ceph-mon[36901]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-16T12:14:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:03 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi098 2024-03-16T12:14:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:03 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi098 to 8545M 2024-03-16T12:14:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:03 smithi173 ceph-mon[36632]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-16T12:14:04.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:04 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/2230545244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:04 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:04 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]': finished 2024-03-16T12:14:04.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:04 smithi098 ceph-mon[31633]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T12:14:04.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:04.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:04 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/2230545244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:04 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:04 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]': finished 2024-03-16T12:14:04.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:04 smithi153 ceph-mon[36901]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T12:14:04.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:04 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:04 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/2230545244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:04 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]: dispatch 2024-03-16T12:14:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:04 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b864c756-758e-4799-a35d-b83d204b3727"}]': finished 2024-03-16T12:14:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:04 smithi173 ceph-mon[36632]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T12:14:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:04 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:05.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:05 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/1510177783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:05.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:05 smithi098 ceph-mon[31633]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:05.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:05 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/1510177783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:05.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:05 smithi153 ceph-mon[36901]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:05 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/1510177783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:05 smithi173 ceph-mon[36632]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:07.889 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:07 smithi153 ceph-mon[36901]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:14:07.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:07 smithi098 ceph-mon[31633]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:14:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:07 smithi173 ceph-mon[36632]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:14:09.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:09 smithi098 ceph-mon[31633]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-16T12:14:10.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:09 smithi153 ceph-mon[36901]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-16T12:14:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:09 smithi173 ceph-mon[36632]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-16T12:14:11.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:11 smithi098 ceph-mon[31633]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:14:11.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T12:14:11.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:12.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:11 smithi153 ceph-mon[36901]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:14:12.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T12:14:12.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:11 smithi173 ceph-mon[36632]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:14:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T12:14:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:12.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:12 smithi153 ceph-mon[36901]: Deploying daemon osd.4 on smithi153 2024-03-16T12:14:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:12 smithi173 ceph-mon[36632]: Deploying daemon osd.4 on smithi153 2024-03-16T12:14:13.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:12 smithi098 ceph-mon[31633]: Deploying daemon osd.4 on smithi153 2024-03-16T12:14:14.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:13 smithi153 ceph-mon[36901]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:13 smithi173 ceph-mon[36632]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:14.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:13 smithi098 ceph-mon[31633]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:15.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:15 smithi153 ceph-mon[36901]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:15.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:15 smithi098 ceph-mon[31633]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:15 smithi173 ceph-mon[36632]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:17.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:17 smithi098 ceph-mon[31633]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:17.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:17.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:17.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:18.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:17 smithi153 ceph-mon[36901]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:18.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:18.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:18.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:17 smithi173 ceph-mon[36632]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:19.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:19 smithi098 ceph-mon[31633]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:19.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:19 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:19 smithi153 ceph-mon[36901]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:20.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:19 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:19 smithi173 ceph-mon[36632]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:19 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.388+0000 7fbf527fc700 1 -- 172.21.15.153:0/2821386964 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf5c10a3b0 con 0x7fbf480a2890 2024-03-16T12:14:20.374 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-03-16T12:14:20.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 msgr2=0x7fbf480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:20.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 0x7fbf480a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00b580 tx=0x7fbf4c005fb0 comp rx=0 tx=0).stop 2024-03-16T12:14:20.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 msgr2=0x7fbf5c1a1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:20.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400b970 tx=0x7fbf5400bc80 comp rx=0 tx=0).stop 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 shutdown_connections 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf480a2890 0x7fbf480a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c103e90 0x7fbf5c1a1e10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf5c1022f0 0x7fbf5c19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 --2- 172.21.15.153:0/2821386964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf5c1034f0 0x7fbf5c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 >> 172.21.15.153:0/2821386964 conn(0x7fbf5c0fd8b0 msgr2=0x7fbf5c108c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 shutdown_connections 2024-03-16T12:14:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:14:20.391+0000 7fbf6384b700 1 -- 172.21.15.153:0/2821386964 wait complete. 2024-03-16T12:14:20.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:20.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:20 smithi098 ceph-mon[31633]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:21.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:21.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:20 smithi153 ceph-mon[36901]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:21.057 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T12:14:21.059 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-03-16T12:14:21.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T12:14:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:21.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:21.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:20 smithi173 ceph-mon[36632]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T12:14:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:14:21.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:21.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T12:14:21.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:21.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T12:14:21.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:21.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:21 smithi098 ceph-mon[31633]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:22.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:21 smithi153 ceph-mon[36901]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: from='osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:21 smithi173 ceph-mon[36632]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:14:23.389 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:14:23.078+0000 7f67f853a700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T12:14:23.390 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:14:23.084+0000 7f67f0325700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132374 s, 792 MB/s 2024-03-16T12:14:23.456 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-16T12:14:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:23 smithi173 ceph-mon[36632]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:14:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:23 smithi173 ceph-mon[36632]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T12:14:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:23 smithi173 ceph-mon[36632]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:23 smithi173 ceph-mon[36632]: from='osd.4 ' entity='osd.4' 2024-03-16T12:14:23.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:23 smithi098 ceph-mon[31633]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:14:23.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:23 smithi098 ceph-mon[31633]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T12:14:23.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:23.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:23 smithi098 ceph-mon[31633]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:23.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:23 smithi098 ceph-mon[31633]: from='osd.4 ' entity='osd.4' 2024-03-16T12:14:23.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-mon[36901]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:14:23.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-mon[36901]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T12:14:23.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:23.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-mon[36901]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-16T12:14:23.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:23 smithi153 ceph-mon[36901]: from='osd.4 ' entity='osd.4' 2024-03-16T12:14:24.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-03-16T12:14:24.340 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:14:24.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:14:24.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:14:24.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] boot 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:24.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] boot 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:24.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:14:24.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: osd.4 [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] boot 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:24.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:25.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:25 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi153 2024-03-16T12:14:25.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:25 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-16T12:14:25.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:25 smithi098 ceph-mon[31633]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:25.710 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:25 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi153 2024-03-16T12:14:25.710 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:25 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-16T12:14:25.710 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:25 smithi173 ceph-mon[36632]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:25.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:25 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi153 2024-03-16T12:14:25.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:25 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi153 to 8540M 2024-03-16T12:14:25.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:25 smithi153 ceph-mon[36901]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1326634325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb648102e80 msgr2=0x7fb648105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1326634325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb648102e80 0x7fb648105270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb63800b0a0 tx=0x7fb63800b3b0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1326634325 shutdown_connections 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1326634325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6481080e0 0x7fb64810a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1326634325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb648102e80 0x7fb648105270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1326634325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6481057b0 0x7fb648107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1326634325 >> 172.21.15.173:0/1326634325 conn(0x7fb6480fc830 msgr2=0x7fb6480fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1326634325 shutdown_connections 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.958+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1326634325 wait complete. 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 Processor -- start 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 -- start start 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648102e80 0x7fb648198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6481080e0 0x7fb64819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810df70 con 0x7fb6481080e0 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810ddf0 con 0x7fb648102e80 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb64d5d6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810e0f0 con 0x7fb6481057b0 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:48856/0 (socket says 172.21.15.173:48856) 2024-03-16T12:14:25.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 -- 172.21.15.173:0/1129508228 learned_addr learned my addr 172.21.15.173:0/1129508228 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb646ffd700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648102e80 0x7fb648198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 -- 172.21.15.173:0/1129508228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648102e80 msgr2=0x7fb648198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648102e80 0x7fb648198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 -- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6481080e0 msgr2=0x7fb64819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6481080e0 0x7fb64819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.959+0000 7fb6467fc700 1 -- 172.21.15.173:0/1129508228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481a0190 con 0x7fb6481057b0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb6467fc700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb63000b560 tx=0x7fb63000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630015020 con 0x7fb6481057b0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb630011d10 con 0x7fb6481057b0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6481a0480 con 0x7fb6481057b0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630010e50 con 0x7fb6481057b0 2024-03-16T12:14:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.960+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481a09c0 con 0x7fb6481057b0 2024-03-16T12:14:25.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.961+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb648068f90 con 0x7fb6481057b0 2024-03-16T12:14:25.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.962+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb630018020 con 0x7fb6481057b0 2024-03-16T12:14:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.963+0000 7fb63ffff700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 0x7fb6340a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.963+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fb6300cd770 con 0x7fb6481057b0 2024-03-16T12:14:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.963+0000 7fb646ffd700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 0x7fb6340a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.964+0000 7fb646ffd700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 0x7fb6340a4ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb638009fd0 tx=0x7fb63800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:25.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:25.967+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb63001f020 con 0x7fb6481057b0 2024-03-16T12:14:26.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:26.167+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb648062710 con 0x7fb6340a2a30 2024-03-16T12:14:26.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:26 smithi098 ceph-mon[31633]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T12:14:26.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:26.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:26.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:26.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:26 smithi153 ceph-mon[36901]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T12:14:26.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:26.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:26.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:26.775 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:26 smithi173 ceph-mon[36632]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T12:14:26.775 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:26.776 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:26.776 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:27.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:27 smithi098 ceph-mon[31633]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:27.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:27 smithi098 ceph-mon[31633]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:27.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:27 smithi153 ceph-mon[36901]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:27.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:27 smithi153 ceph-mon[36901]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:27.784 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:27 smithi173 ceph-mon[36632]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:27.784 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:27 smithi173 ceph-mon[36632]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:29.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/2998003802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]': finished 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:29.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/2626457877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/2998003802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]': finished 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:30.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/2626457877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/2998003802' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]: dispatch 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c31daad0-fd36-4b7c-98d3-64e2057c4779"}]': finished 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/2626457877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:14:31.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:31 smithi098 ceph-mon[31633]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:32.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:31 smithi153 ceph-mon[36901]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:31 smithi173 ceph-mon[36632]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:33.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:33 smithi098 ceph-mon[31633]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:34.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:33 smithi153 ceph-mon[36901]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:34.013 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:33 smithi173 ceph-mon[36632]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:36.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:35 smithi153 ceph-mon[36901]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:35 smithi173 ceph-mon[36632]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:36.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:35 smithi098 ceph-mon[31633]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:37.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T12:14:37.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:37.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T12:14:37.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:37.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T12:14:37.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:38.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:37 smithi173 ceph-mon[36632]: Deploying daemon osd.5 on smithi173 2024-03-16T12:14:38.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:37 smithi173 ceph-mon[36632]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:38.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:37 smithi098 ceph-mon[31633]: Deploying daemon osd.5 on smithi173 2024-03-16T12:14:38.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:37 smithi098 ceph-mon[31633]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:38.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:37 smithi153 ceph-mon[36901]: Deploying daemon osd.5 on smithi173 2024-03-16T12:14:38.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:37 smithi153 ceph-mon[36901]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:39 smithi173 ceph-mon[36632]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:40.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:39 smithi098 ceph-mon[31633]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:40.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:39 smithi153 ceph-mon[36901]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:42.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:41 smithi173 ceph-mon[36632]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:42.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:41 smithi098 ceph-mon[31633]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:42.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:41 smithi153 ceph-mon[36901]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:43 smithi173 ceph-mon[36632]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:43.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:43.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:43 smithi098 ceph-mon[31633]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:43.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:43.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:43.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:43 smithi153 ceph-mon[36901]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:45.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:45.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:45.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:45.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:45.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:45.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:45 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:45.986 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-03-16T12:14:45.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.001+0000 7fb63ffff700 1 -- 172.21.15.173:0/1129508228 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb648062710 con 0x7fb6340a2a30 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 msgr2=0x7fb6340a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 0x7fb6340a4ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb638009fd0 tx=0x7fb63800b7b0 comp rx=0 tx=0).stop 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 msgr2=0x7fb648198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb63000b560 tx=0x7fb63000b870 comp rx=0 tx=0).stop 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 shutdown_connections 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6340a2a30 0x7fb6340a4ef0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648102e80 0x7fb648198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6481080e0 0x7fb64819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 --2- 172.21.15.173:0/1129508228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481057b0 0x7fb648198a70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 >> 172.21.15.173:0/1129508228 conn(0x7fb6480fc830 msgr2=0x7fb648103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:14:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 shutdown_connections 2024-03-16T12:14:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:14:46.003+0000 7fb64d5d6700 1 -- 172.21.15.173:0/1129508228 wait complete. 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:45.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:45 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:46.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:45 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.702 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T12:14:46.704 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-16T12:14:46.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:46 smithi173 ceph-mon[36632]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:46 smithi173 ceph-mon[36632]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:46 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:46 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:46 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:46 smithi098 ceph-mon[31633]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:46.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:46 smithi098 ceph-mon[31633]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:46.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:46 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:46 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:46.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:46 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:46 smithi153 ceph-mon[36901]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:46 smithi153 ceph-mon[36901]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:46 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:46 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:46 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:14:47.143 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:14:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T12:14:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T12:14:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:47 smithi173 ceph-mon[36632]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:47 smithi098 ceph-mon[31633]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: from='osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:48.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:47 smithi153 ceph-mon[36901]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:14:48.761 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:14:48.472+0000 7f6031b6b700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T12:14:48.762 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:14:48.480+0000 7f6029956700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:14:49.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:14:49.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T12:14:49.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: from='osd.5 ' entity='osd.5' 2024-03-16T12:14:49.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] boot 2024-03-16T12:14:49.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:48 smithi173 ceph-mon[36632]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T12:14:49.103 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:14:49.103 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T12:14:49.103 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.103 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.104 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: from='osd.5 ' entity='osd.5' 2024-03-16T12:14:49.104 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] boot 2024-03-16T12:14:49.104 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:48 smithi098 ceph-mon[31633]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: from='osd.5 ' entity='osd.5' 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: osd.5 [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] boot 2024-03-16T12:14:49.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:48 smithi153 ceph-mon[36901]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T12:14:50.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:49 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:14:50.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:49 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:14:50.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:49 smithi173 ceph-mon[36632]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:50.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:49 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:50.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:49 smithi173 ceph-mon[36632]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T12:14:50.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:49 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:14:50.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:49 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:14:50.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:49 smithi098 ceph-mon[31633]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:50.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:49 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:50.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:49 smithi098 ceph-mon[31633]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.06261 s, 167 MB/s 2024-03-16T12:14:50.162 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T12:14:50.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:49 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:14:50.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:49 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:14:50.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:49 smithi153 ceph-mon[36901]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-16T12:14:50.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:49 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T12:14:50.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:49 smithi153 ceph-mon[36901]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T12:14:51.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi173 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:51.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:51.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:14:51.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:51 smithi098 ceph-mon[31633]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi173 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:51.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:51.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:14:51.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:51 smithi153 ceph-mon[36901]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T12:14:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi173 2024-03-16T12:14:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi173 to 8540M 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:14:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:51 smithi173 ceph-mon[36632]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T12:14:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:14:52.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:52 smithi098 ceph-mon[31633]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T12:14:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:52 smithi173 ceph-mon[36632]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T12:14:53.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:52 smithi153 ceph-mon[36901]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T12:14:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 -- 172.21.15.98:0/3880737392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d010a1d0 msgr2=0x7f70d010c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 --2- 172.21.15.98:0/3880737392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d010a1d0 0x7f70d010c660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b0a0 tx=0x7f70c800b3b0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 -- 172.21.15.98:0/3880737392 shutdown_connections 2024-03-16T12:14:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 --2- 172.21.15.98:0/3880737392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d010a1d0 0x7f70d010c660 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 --2- 172.21.15.98:0/3880737392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d0076300 0x7f70d0074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 --2- 172.21.15.98:0/3880737392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 0x7f70d00751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 -- 172.21.15.98:0/3880737392 >> 172.21.15.98:0/3880737392 conn(0x7f70d00fd8e0 msgr2=0x7f70d00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:14:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 -- 172.21.15.98:0/3880737392 shutdown_connections 2024-03-16T12:14:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.035+0000 7f70d4c00700 1 -- 172.21.15.98:0/3880737392 wait complete. 2024-03-16T12:14:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.036+0000 7f70d4c00700 1 Processor -- start 2024-03-16T12:14:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 -- start start 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 0x7f70d019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 0x7f70d01a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01100f0 con 0x7f70d0076300 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70d010ff70 con 0x7f70d0074d30 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70d4c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0110270 con 0x7f70d010a1d0 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70cd400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70cde00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 0x7f70d019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70cd400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59124/0 (socket says 172.21.15.98:59124) 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.037+0000 7f70cd400700 1 -- 172.21.15.98:0/268036922 learned_addr learned my addr 172.21.15.98:0/268036922 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70ce800700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 0x7f70d01a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 -- 172.21.15.98:0/268036922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 msgr2=0x7f70d01a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 0x7f70d01a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 -- 172.21.15.98:0/268036922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 msgr2=0x7f70d019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 0x7f70d019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 -- 172.21.15.98:0/268036922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d01a2330 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70ce800700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 0x7f70d01a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70cd400700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00e9a0 tx=0x7f70bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc00cbd0 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d01a2620 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70bc004500 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.038+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc003e00 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.039+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d01a2b60 con 0x7f70d0076300 2024-03-16T12:14:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.040+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0068f90 con 0x7f70d0076300 2024-03-16T12:14:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.042+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f70bc003fa0 con 0x7f70d0076300 2024-03-16T12:14:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.043+0000 7f70c6a00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 0x7f70b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:14:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.043+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f70bc0ce010 con 0x7f70d0076300 2024-03-16T12:14:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.043+0000 7f70cde00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 0x7f70b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:14:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.044+0000 7f70cde00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 0x7f70b40a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70d00766e0 tx=0x7f70c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:14:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.046+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f70bc01f340 con 0x7f70d0076300 2024-03-16T12:14:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:53 smithi173 ceph-mon[36632]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T12:14:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:53 smithi173 ceph-mon[36632]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T12:14:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:53 smithi098 ceph-mon[31633]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T12:14:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:53 smithi098 ceph-mon[31633]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T12:14:54.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:53 smithi153 ceph-mon[36901]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 394 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T12:14:54.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:53 smithi153 ceph-mon[36901]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T12:14:54.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:14:54.278+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f70d0062710 con 0x7f70b40a2890 2024-03-16T12:14:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:55.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:55.175 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:55.175 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:55.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:14:55.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:14:55.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:14:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:55 smithi173 ceph-mon[36632]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:55 smithi173 ceph-mon[36632]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-16T12:14:56.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:55 smithi098 ceph-mon[31633]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:56.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:55 smithi098 ceph-mon[31633]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-16T12:14:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:55 smithi153 ceph-mon[36901]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:14:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:55 smithi153 ceph-mon[36901]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-16T12:14:57.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:57 smithi098 ceph-mon[31633]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:57 smithi173 ceph-mon[36632]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:58.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:57 smithi153 ceph-mon[36901]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:14:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:58 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1163505843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:58 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:58 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]': finished 2024-03-16T12:14:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:58 smithi173 ceph-mon[36632]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T12:14:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:14:59.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:58 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1163505843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:58 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:58 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]': finished 2024-03-16T12:14:59.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:58 smithi098 ceph-mon[31633]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T12:14:59.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:14:59.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:58 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1163505843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:58 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]: dispatch 2024-03-16T12:14:59.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:58 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "229b1cd5-870e-4b6c-b3af-243b19a68157"}]': finished 2024-03-16T12:14:59.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:58 smithi153 ceph-mon[36901]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T12:14:59.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:59 smithi173 ceph-mon[36632]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:14:59 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4137998963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:00.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:59 smithi098 ceph-mon[31633]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:00.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:14:59 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4137998963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:00.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:59 smithi153 ceph-mon[36901]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:00.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:14:59 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4137998963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:01 smithi173 ceph-mon[36632]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:02.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:01 smithi098 ceph-mon[31633]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:02.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:01 smithi153 ceph-mon[36901]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:03 smithi173 ceph-mon[36632]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:04.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:03 smithi098 ceph-mon[31633]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:03 smithi153 ceph-mon[36901]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:06.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:05 smithi173 ceph-mon[36632]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:06.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:05 smithi098 ceph-mon[31633]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:06.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:05 smithi153 ceph-mon[36901]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:08.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:07 smithi098 ceph-mon[31633]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:08.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:07 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T12:15:08.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:07 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:08.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:07 smithi153 ceph-mon[36901]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:08.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:07 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T12:15:08.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:07 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:07 smithi173 ceph-mon[36632]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:08.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:07 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T12:15:08.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:07 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:09.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:08 smithi098 ceph-mon[31633]: Deploying daemon osd.6 on smithi098 2024-03-16T12:15:09.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:08 smithi153 ceph-mon[36901]: Deploying daemon osd.6 on smithi098 2024-03-16T12:15:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:08 smithi173 ceph-mon[36632]: Deploying daemon osd.6 on smithi098 2024-03-16T12:15:10.175 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:09 smithi098 ceph-mon[31633]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:10.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:09 smithi153 ceph-mon[36901]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:09 smithi173 ceph-mon[36632]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:11.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:11 smithi098 ceph-mon[31633]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:12.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:11 smithi153 ceph-mon[36901]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:12.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:11 smithi173 ceph-mon[36632]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:14.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:13 smithi098 ceph-mon[31633]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:14.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:13 smithi153 ceph-mon[36901]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:13 smithi173 ceph-mon[36632]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:15.926 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:15 smithi098 ceph-mon[31633]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:15 smithi153 ceph-mon[36901]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:15 smithi173 ceph-mon[36632]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:17.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:17.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:17 smithi098 ceph-mon[31633]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:17.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:17.507 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:17 smithi153 ceph-mon[36901]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:17.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:17.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:17 smithi173 ceph-mon[36632]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:20.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:19 smithi153 ceph-mon[36901]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:20.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:19 smithi098 ceph-mon[31633]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:19 smithi173 ceph-mon[36632]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:20.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:20 smithi098 ceph-mon[31633]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T12:15:20.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:20.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:21.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:20 smithi153 ceph-mon[36901]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T12:15:21.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:21.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:21.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.063+0000 7f70c6a00700 1 -- 172.21.15.98:0/268036922 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f70d0062710 con 0x7f70b40a2890 2024-03-16T12:15:21.050 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-03-16T12:15:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 msgr2=0x7f70b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 0x7f70b40a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f70d00766e0 tx=0x7f70c0009260 comp rx=0 tx=0).stop 2024-03-16T12:15:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 msgr2=0x7f70d019ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00e9a0 tx=0x7f70bc00ecb0 comp rx=0 tx=0).stop 2024-03-16T12:15:21.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 shutdown_connections 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70b40a2890 0x7f70b40a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d0074d30 0x7f70d019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70d010a1d0 0x7f70d01a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 --2- 172.21.15.98:0/268036922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d0076300 0x7f70d019ac10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 >> 172.21.15.98:0/268036922 conn(0x7f70d00fd8e0 msgr2=0x7f70d01060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.067+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 shutdown_connections 2024-03-16T12:15:21.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:15:21.068+0000 7f70d4c00700 1 -- 172.21.15.98:0/268036922 wait complete. 2024-03-16T12:15:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:20 smithi173 ceph-mon[36632]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T12:15:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:21.905 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T12:15:21.910 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_2... 2024-03-16T12:15:21.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.156 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:22.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:22.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-mon[31633]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:15:22.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-mon[31633]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T12:15:22.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:22.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:22.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-mon[31633]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' 2024-03-16T12:15:22.928 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:15:22.595+0000 7fdb85400700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T12:15:22.928 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:15:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:15:22.629+0000 7fdb7b200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:15:23.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:22 smithi153 ceph-mon[36901]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:15:23.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:22 smithi153 ceph-mon[36901]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T12:15:23.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:23.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:23.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:22 smithi153 ceph-mon[36901]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' 2024-03-16T12:15:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:22 smithi173 ceph-mon[36632]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:15:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:22 smithi173 ceph-mon[36632]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T12:15:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:22 smithi173 ceph-mon[36632]: from='osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753]' entity='osd.6' 2024-03-16T12:15:23.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] boot 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T12:15:23.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] boot 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T12:15:24.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: osd.6 [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] boot 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T12:15:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T12:15:24.175 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T12:15:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T12:15:24.176 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T12:15:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T12:15:24.176 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122834 s, 854 MB/s 2024-03-16T12:15:24.176 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T12:15:24.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-03-16T12:15:25.102 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:15:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:25 smithi173 ceph-mon[36632]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:25 smithi173 ceph-mon[36632]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T12:15:26.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:25 smithi098 ceph-mon[31633]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:26.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:25 smithi098 ceph-mon[31633]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T12:15:26.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:25 smithi153 ceph-mon[36901]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:26.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:25 smithi153 ceph-mon[36901]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 -- 172.21.15.153:0/1013743561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 msgr2=0x7fb630102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 --2- 172.21.15.153:0/1013743561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630102890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0099c0 tx=0x7fb61c009cd0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 -- 172.21.15.153:0/1013743561 shutdown_connections 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 --2- 172.21.15.153:0/1013743561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 0x7fb630109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 --2- 172.21.15.153:0/1013743561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 0x7fb630101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 --2- 172.21.15.153:0/1013743561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630102890 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 -- 172.21.15.153:0/1013743561 >> 172.21.15.153:0/1013743561 conn(0x7fb6300fc7d0 msgr2=0x7fb6300fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 -- 172.21.15.153:0/1013743561 shutdown_connections 2024-03-16T12:15:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.811+0000 7fb6371e0700 1 -- 172.21.15.153:0/1013743561 wait complete. 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 Processor -- start 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 -- start start 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 0x7fb630197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 0x7fb63019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb63010cf10 con 0x7fb630102dd0 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb63010cd90 con 0x7fb630102430 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb6371e0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb63010d090 con 0x7fb630101230 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50722/0 (socket says 172.21.15.153:50722) 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 -- 172.21.15.153:0/3218891995 learned_addr learned my addr 172.21.15.153:0/3218891995 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T12:15:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb634f7c700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 0x7fb630197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb63577d700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 0x7fb63019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 -- 172.21.15.153:0/3218891995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 msgr2=0x7fb630197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 0x7fb630197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 -- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 msgr2=0x7fb63019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 0x7fb63019eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.812+0000 7fb62ffff700 1 -- 172.21.15.153:0/3218891995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63006aa70 con 0x7fb630102430 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb62ffff700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb61c003890 tx=0x7fb61c003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c027070 con 0x7fb630102430 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63006ad00 con 0x7fb630102430 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb61c022470 con 0x7fb630102430 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c00f6c0 con 0x7fb630102430 2024-03-16T12:15:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.813+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63006b2b0 con 0x7fb630102430 2024-03-16T12:15:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.814+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb630068f90 con 0x7fb630102430 2024-03-16T12:15:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.815+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb61c02a070 con 0x7fb630102430 2024-03-16T12:15:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.816+0000 7fb62dffb700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 0x7fb6200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:26.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.816+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fb61c0cee20 con 0x7fb630102430 2024-03-16T12:15:26.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.816+0000 7fb634f7c700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 0x7fb6200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:26.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.817+0000 7fb634f7c700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 0x7fb6200a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb624006fd0 tx=0x7fb624008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:15:26.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:26.821+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb61c05e740 con 0x7fb630102430 2024-03-16T12:15:27.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:27.024+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb630062710 con 0x7fb6200a2890 2024-03-16T12:15:27.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:27 smithi153 ceph-mon[36901]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:27.895 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:27.895 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:27.895 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:27.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:27 smithi098 ceph-mon[31633]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:27.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:27.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:27.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:27 smithi173 ceph-mon[36632]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:28.970 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:28.970 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi098 2024-03-16T12:15:28.970 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:28.970 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi098 to 5697M 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:28.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi098 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi098 to 5697M 2024-03-16T12:15:29.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:29.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi098 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi098 to 5697M 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:30.177 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:30.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/2486493819' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]': finished 2024-03-16T12:15:30.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T12:15:30.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:29 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:30.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:30.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/2486493819' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]': finished 2024-03-16T12:15:30.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T12:15:30.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:29 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:30.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/2486493819' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]: dispatch 2024-03-16T12:15:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615a84ba-ee1b-4ea0-92c9-0211098d2215"}]': finished 2024-03-16T12:15:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T12:15:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:29 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:31.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:30 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/3095151039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:31.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:30 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/3095151039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:31.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:30 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/3095151039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:32.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:31 smithi098 ceph-mon[31633]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:32.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:31 smithi153 ceph-mon[36901]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:32.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:31 smithi173 ceph-mon[36632]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:34.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:33 smithi153 ceph-mon[36901]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:34.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:33 smithi098 ceph-mon[31633]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:34.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:33 smithi173 ceph-mon[36632]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:36.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:35 smithi153 ceph-mon[36901]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:36.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:35 smithi098 ceph-mon[31633]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:35 smithi173 ceph-mon[36632]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:38.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:37 smithi153 ceph-mon[36901]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:38.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T12:15:38.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:37 smithi098 ceph-mon[31633]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T12:15:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:37 smithi173 ceph-mon[36632]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T12:15:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:39.161 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:38 smithi153 ceph-mon[36901]: Deploying daemon osd.7 on smithi153 2024-03-16T12:15:39.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:38 smithi098 ceph-mon[31633]: Deploying daemon osd.7 on smithi153 2024-03-16T12:15:39.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:38 smithi173 ceph-mon[36632]: Deploying daemon osd.7 on smithi153 2024-03-16T12:15:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:39 smithi098 ceph-mon[31633]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:39 smithi153 ceph-mon[36901]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:40.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:39 smithi173 ceph-mon[36632]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:42.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:41 smithi153 ceph-mon[36901]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:41 smithi098 ceph-mon[31633]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:42.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:41 smithi173 ceph-mon[36632]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:43.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:43 smithi153 ceph-mon[36901]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:43.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:43.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:43.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:44.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:43 smithi098 ceph-mon[31633]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:44.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:44.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:44.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:43 smithi173 ceph-mon[36632]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:45.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:45 smithi153 ceph-mon[36901]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:46.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:45 smithi098 ceph-mon[31633]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:46.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:45 smithi173 ceph-mon[36632]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:46.592 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-03-16T12:15:46.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.608+0000 7fb62dffb700 1 -- 172.21.15.153:0/3218891995 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb630062710 con 0x7fb6200a2890 2024-03-16T12:15:46.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 msgr2=0x7fb6200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:46.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 0x7fb6200a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb624006fd0 tx=0x7fb624008040 comp rx=0 tx=0).stop 2024-03-16T12:15:46.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 msgr2=0x7fb630197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:46.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb61c003890 tx=0x7fb61c003730 comp rx=0 tx=0).stop 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 shutdown_connections 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb630102dd0 0x7fb63019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb630101230 0x7fb630197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6200a2890 0x7fb6200a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 --2- 172.21.15.153:0/3218891995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb630102430 0x7fb630197aa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 >> 172.21.15.153:0/3218891995 conn(0x7fb6300fc7d0 msgr2=0x7fb6301057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 shutdown_connections 2024-03-16T12:15:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:15:46.610+0000 7fb6371e0700 1 -- 172.21.15.153:0/3218891995 wait complete. 2024-03-16T12:15:47.215 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T12:15:47.217 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_2... 2024-03-16T12:15:47.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T12:15:47.465 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:47.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:48.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:48 smithi173 ceph-mon[36632]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T12:15:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:48 smithi173 ceph-mon[36632]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:48 smithi173 ceph-mon[36632]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T12:15:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:48 smithi173 ceph-mon[36632]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:48.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-mon[36901]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T12:15:48.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-mon[36901]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:48.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-mon[36901]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T12:15:48.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:48.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-mon[36901]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:48.993 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:15:48.583+0000 7fd2d2397700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T12:15:48.994 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:15:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:15:48.594+0000 7fd2ce18a700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:15:48.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:48 smithi098 ceph-mon[31633]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T12:15:48.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:48 smithi098 ceph-mon[31633]: from='osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:48.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:48 smithi098 ceph-mon[31633]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T12:15:48.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:48.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:48 smithi098 ceph-mon[31633]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111271 s, 942 MB/s 2024-03-16T12:15:49.519 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-16T12:15:49.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:49 smithi153 ceph-mon[36901]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:15:49.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:49 smithi153 ceph-mon[36901]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T12:15:49.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:49 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:49.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:49 smithi153 ceph-mon[36901]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:49.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:49 smithi173 ceph-mon[36632]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:15:49.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:49 smithi173 ceph-mon[36632]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T12:15:49.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:49 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:49.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:49 smithi173 ceph-mon[36632]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:49.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:49 smithi098 ceph-mon[31633]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:15:49.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:49 smithi098 ceph-mon[31633]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T12:15:49.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:49 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:49.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:49 smithi098 ceph-mon[31633]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-16T12:15:50.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-03-16T12:15:50.447 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:15:50.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:15:50.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:15:50.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] boot 2024-03-16T12:15:50.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T12:15:50.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:50.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:50 smithi153 ceph-mon[36901]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T12:15:50.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:15:50.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:15:50.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] boot 2024-03-16T12:15:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T12:15:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:50 smithi173 ceph-mon[36632]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: osd.7 [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] boot 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T12:15:50.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:50 smithi098 ceph-mon[31633]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi153 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T12:15:52.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:51 smithi173 ceph-mon[36632]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 -- 172.21.15.173:0/613825941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 msgr2=0x7efc98107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 --2- 172.21.15.173:0/613825941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc98107ab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efc84009990 tx=0x7efc84009ca0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 -- 172.21.15.173:0/613825941 shutdown_connections 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 --2- 172.21.15.173:0/613825941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 0x7efc98105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 --2- 172.21.15.173:0/613825941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc98107ff0 0x7efc9810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 --2- 172.21.15.173:0/613825941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc98107ab0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 -- 172.21.15.173:0/613825941 >> 172.21.15.173:0/613825941 conn(0x7efc980fc7c0 msgr2=0x7efc980febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 -- 172.21.15.173:0/613825941 shutdown_connections 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 -- 172.21.15.173:0/613825941 wait complete. 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.223+0000 7efc9f167700 1 Processor -- start 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 -- start start 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 0x7efc9819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc98107ff0 0x7efc981a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810df10 con 0x7efc98107ff0 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810dd90 con 0x7efc98102d90 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810e090 con 0x7efc981056c0 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:44666/0 (socket says 172.21.15.173:44666) 2024-03-16T12:15:52.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 -- 172.21.15.173:0/210679459 learned_addr learned my addr 172.21.15.173:0/210679459 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9cf03700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 0x7efc9819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 -- 172.21.15.173:0/210679459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 msgr2=0x7efc9819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 0x7efc9819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 -- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc98107ff0 msgr2=0x7efc981a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc98107ff0 0x7efc981a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 -- 172.21.15.173:0/210679459 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981a23f0 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc97fff700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc84005f50 tx=0x7efc8400bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84026070 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981a2680 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc8400f460 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.224+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84021670 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.225+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981a2b30 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.225+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc98068f90 con 0x7efc981056c0 2024-03-16T12:15:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.227+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efc84029070 con 0x7efc981056c0 2024-03-16T12:15:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.227+0000 7efc95ffb700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 0x7efc880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:15:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.228+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7efc840cf2c0 con 0x7efc981056c0 2024-03-16T12:15:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.228+0000 7efc9cf03700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 0x7efc880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:15:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.229+0000 7efc9cf03700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 0x7efc880a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009c60 tx=0x7efc8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:15:52.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.231+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc84030020 con 0x7efc981056c0 2024-03-16T12:15:52.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi153 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T12:15:52.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:51 smithi098 ceph-mon[31633]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-16T12:15:52.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:52.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi153 2024-03-16T12:15:52.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T12:15:52.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:51 smithi153 ceph-mon[36901]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-16T12:15:52.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:15:52.438+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7efc98062710 con 0x7efc880a2890 2024-03-16T12:15:53.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:52 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-16T12:15:53.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:52 smithi098 ceph-mon[31633]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T12:15:53.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:52 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:53.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:52 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:53.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:52 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:53.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:52 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-16T12:15:53.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:52 smithi153 ceph-mon[36901]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T12:15:53.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:53.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:53.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:52 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:53.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:52 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi153 to 5693M 2024-03-16T12:15:53.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:52 smithi173 ceph-mon[36632]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T12:15:53.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:52 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:15:53.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:52 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:15:53.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:52 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:15:54.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:53 smithi173 ceph-mon[36632]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:54.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:53 smithi173 ceph-mon[36632]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:54.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:53 smithi173 ceph-mon[36632]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T12:15:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:53 smithi098 ceph-mon[31633]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:53 smithi098 ceph-mon[31633]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:53 smithi098 ceph-mon[31633]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T12:15:54.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:53 smithi153 ceph-mon[36901]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:15:54.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:53 smithi153 ceph-mon[36901]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:54.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:53 smithi153 ceph-mon[36901]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T12:15:55.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:55 smithi153 ceph-mon[36901]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:55.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:55 smithi153 ceph-mon[36901]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-16T12:15:55.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:55 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:15:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:55 smithi173 ceph-mon[36632]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:55 smithi173 ceph-mon[36632]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-16T12:15:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:55 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:15:55.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:55 smithi098 ceph-mon[31633]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:55.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:55 smithi098 ceph-mon[31633]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-16T12:15:55.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:55 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:15:56.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/2115729712' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]': finished 2024-03-16T12:15:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T12:15:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:15:56.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:56 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/3242607614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/2115729712' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]': finished 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:15:56.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:56 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/3242607614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:56.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/2115729712' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]: dispatch 2024-03-16T12:15:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa255236-2797-4a72-b784-65cc6c731190"}]': finished 2024-03-16T12:15:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T12:15:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:15:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/3242607614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:15:57.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:57 smithi173 ceph-mon[36632]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:57.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:57 smithi098 ceph-mon[31633]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:57.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:57 smithi153 ceph-mon[36901]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:15:59.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:15:59 smithi173 ceph-mon[36632]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:15:59 smithi098 ceph-mon[31633]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:00.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:15:59 smithi153 ceph-mon[36901]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:01.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:01 smithi173 ceph-mon[36632]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:02.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:01 smithi098 ceph-mon[31633]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:01 smithi153 ceph-mon[36901]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:03.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T12:16:03.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:02 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:03.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T12:16:03.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:02 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:03.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T12:16:03.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:02 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:03.982 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:03 smithi173 ceph-mon[36632]: Deploying daemon osd.8 on smithi173 2024-03-16T12:16:03.982 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:03 smithi173 ceph-mon[36632]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:04.157 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:03 smithi153 ceph-mon[36901]: Deploying daemon osd.8 on smithi173 2024-03-16T12:16:04.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:03 smithi153 ceph-mon[36901]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:04.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:03 smithi098 ceph-mon[31633]: Deploying daemon osd.8 on smithi173 2024-03-16T12:16:04.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:03 smithi098 ceph-mon[31633]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:06.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:05 smithi173 ceph-mon[36632]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:06.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:05 smithi098 ceph-mon[31633]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:06.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:05 smithi153 ceph-mon[36901]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:07 smithi173 ceph-mon[36632]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:07 smithi098 ceph-mon[31633]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:08.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:07 smithi153 ceph-mon[36901]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:09.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:09.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:09 smithi098 ceph-mon[31633]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:09.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:09.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:09 smithi153 ceph-mon[36901]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:09.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:09 smithi173 ceph-mon[36632]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:11 smithi173 ceph-mon[36632]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:11.956 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-03-16T12:16:11.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.971+0000 7efc95ffb700 1 -- 172.21.15.173:0/210679459 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efc98062710 con 0x7efc880a2890 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 msgr2=0x7efc880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 0x7efc880a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009c60 tx=0x7efc8c009260 comp rx=0 tx=0).stop 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 msgr2=0x7efc9819acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc84005f50 tx=0x7efc8400bbb0 comp rx=0 tx=0).stop 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 shutdown_connections 2024-03-16T12:16:11.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc880a2890 0x7efc880a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc98102d90 0x7efc9819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc98107ff0 0x7efc981a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 --2- 172.21.15.173:0/210679459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc981056c0 0x7efc9819acd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 >> 172.21.15.173:0/210679459 conn(0x7efc980fc7c0 msgr2=0x7efc981023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 shutdown_connections 2024-03-16T12:16:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:16:11.975+0000 7efc9f167700 1 -- 172.21.15.173:0/210679459 wait complete. 2024-03-16T12:16:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:11 smithi098 ceph-mon[31633]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:12.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:11 smithi153 ceph-mon[36901]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:12.685 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T12:16:12.687 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-16T12:16:12.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T12:16:13.133 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:16:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:13.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T12:16:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T12:16:14.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:14.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:13 smithi098 ceph-mon[31633]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:14.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:13 smithi153 ceph-mon[36901]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T12:16:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: from='osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T12:16:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-mon[36632]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:16:14.323 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:16:13.979+0000 7fc1bce0c700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T12:16:14.323 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:16:13 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:16:13.989+0000 7fc1b8bff700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:16:15.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:14 smithi173 ceph-mon[36632]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:16:15.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:14 smithi173 ceph-mon[36632]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T12:16:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:14 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:14 smithi173 ceph-mon[36632]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:15.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:14 smithi098 ceph-mon[31633]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:16:15.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:14 smithi098 ceph-mon[31633]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T12:16:15.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:14 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:15.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:14 smithi098 ceph-mon[31633]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:15.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:14 smithi153 ceph-mon[36901]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:16:15.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:14 smithi153 ceph-mon[36901]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T12:16:15.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:14 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:15.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:14 smithi153 ceph-mon[36901]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-16T12:16:16.170 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:16:16.170 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:16:16.170 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] boot 2024-03-16T12:16:16.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T12:16:16.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:16.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:15 smithi098 ceph-mon[31633]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T12:16:16.171 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T12:16:16.171 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T12:16:16.171 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-16T12:16:16.171 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-16T12:16:16.171 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0311737 s, 336 MB/s 2024-03-16T12:16:16.172 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-16T12:16:16.206 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:16:16.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:16:16.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] boot 2024-03-16T12:16:16.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T12:16:16.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:16.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:15 smithi173 ceph-mon[36632]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: osd.8 [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] boot 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T12:16:16.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:15 smithi153 ceph-mon[36901]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T12:16:17.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi173 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-16T12:16:17.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:17.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:17 smithi098 ceph-mon[31633]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T12:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:16:17.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi173 2024-03-16T12:16:17.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:17 smithi173 ceph-mon[36632]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T12:16:17.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi173 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi173 to 5693M 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:17.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:17.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:17.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:17 smithi153 ceph-mon[36901]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T12:16:18.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:18 smithi098 ceph-mon[31633]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T12:16:18.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:18 smithi153 ceph-mon[36901]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T12:16:18.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:18 smithi173 ceph-mon[36632]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T12:16:19.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:19 smithi098 ceph-mon[31633]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T12:16:19.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:19 smithi098 ceph-mon[31633]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:19.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:19 smithi153 ceph-mon[36901]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T12:16:19.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:19 smithi153 ceph-mon[36901]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 -- 172.21.15.98:0/4166179028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 msgr2=0x7f44cc108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 --2- 172.21.15.98:0/4166179028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc108d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44c00099c0 tx=0x7f44c0009cd0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 -- 172.21.15.98:0/4166179028 shutdown_connections 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 --2- 172.21.15.98:0/4166179028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc108d70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 --2- 172.21.15.98:0/4166179028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 0x7f44cc110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 --2- 172.21.15.98:0/4166179028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 -- 172.21.15.98:0/4166179028 >> 172.21.15.98:0/4166179028 conn(0x7f44cc077920 msgr2=0x7f44cc079d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 -- 172.21.15.98:0/4166179028 shutdown_connections 2024-03-16T12:16:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.827+0000 7f44d4400700 1 -- 172.21.15.98:0/4166179028 wait complete. 2024-03-16T12:16:19.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.829+0000 7f44d4400700 1 Processor -- start 2024-03-16T12:16:19.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.829+0000 7f44d4400700 1 -- start start 2024-03-16T12:16:19.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 0x7f44cc1a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114580 con 0x7f44cc10a4f0 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114400 con 0x7f44cc108950 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d4400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc114700 con 0x7f44cc109b50 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d1000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d1000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42372/0 (socket says 172.21.15.98:42372) 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d1000700 1 -- 172.21.15.98:0/3140634314 learned_addr learned my addr 172.21.15.98:0/3140634314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d2400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 0x7f44cc1a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.830+0000 7f44d1a00700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 -- 172.21.15.98:0/3140634314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 msgr2=0x7f44cc19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:19.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 -- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 msgr2=0x7f44cc1a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 0x7f44cc1a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 -- 172.21.15.98:0/3140634314 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc1a67d0 con 0x7f44cc109b50 2024-03-16T12:16:19.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1a00700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d1000700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44b800e9f0 tx=0x7f44b800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800cbd0 con 0x7f44cc109b50 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b800ee90 con 0x7f44cc109b50 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc1a6ac0 con 0x7f44cc109b50 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b80056c0 con 0x7f44cc109b50 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.831+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc1a7000 con 0x7f44cc109b50 2024-03-16T12:16:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.832+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44cc068f90 con 0x7f44cc109b50 2024-03-16T12:16:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.834+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f44b8004750 con 0x7f44cc109b50 2024-03-16T12:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.836+0000 7f44ca000700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 0x7f44b40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.836+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f44b80ce340 con 0x7f44cc109b50 2024-03-16T12:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.836+0000 7f44d1a00700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 0x7f44b40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.836+0000 7f44d1a00700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 0x7f44b40a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f44c00038c0 tx=0x7f44c0005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:16:19.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:19 smithi173 ceph-mon[36632]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T12:16:19.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:19 smithi173 ceph-mon[36632]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:19.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:19.840+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44b809bcf0 con 0x7f44cc109b50 2024-03-16T12:16:20.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:20.077+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f44cc062710 con 0x7f44b40a2910 2024-03-16T12:16:20.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:20.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:20.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:20.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:20.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:20.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:20.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:20.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:20.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:21.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:21 smithi098 ceph-mon[31633]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:21.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:21 smithi098 ceph-mon[31633]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T12:16:21.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:21 smithi153 ceph-mon[36901]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:21.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:21 smithi153 ceph-mon[36901]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T12:16:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:21 smithi173 ceph-mon[36632]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:21 smithi173 ceph-mon[36632]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T12:16:24.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:23 smithi173 ceph-mon[36632]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:16:24.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:23 smithi098 ceph-mon[31633]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:16:24.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:23 smithi153 ceph-mon[36901]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:16:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:24 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3980248244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:24 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:24 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]': finished 2024-03-16T12:16:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:24 smithi098 ceph-mon[31633]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T12:16:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:24 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3980248244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:24 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:24 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]': finished 2024-03-16T12:16:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:24 smithi173 ceph-mon[36632]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T12:16:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:25.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:24 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3980248244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:25.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:24 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]: dispatch 2024-03-16T12:16:25.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:24 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc2c29f0-167c-40d8-8ca6-4758e66f435c"}]': finished 2024-03-16T12:16:25.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:24 smithi153 ceph-mon[36901]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T12:16:25.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:26.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:25 smithi173 ceph-mon[36632]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:16:26.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:25 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4162790396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:16:26.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:25 smithi098 ceph-mon[31633]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:16:26.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:25 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4162790396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:16:26.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:25 smithi153 ceph-mon[36901]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:16:26.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:25 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4162790396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:16:28.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:27 smithi173 ceph-mon[36632]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-16T12:16:28.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:27 smithi098 ceph-mon[31633]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-16T12:16:28.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:27 smithi153 ceph-mon[36901]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-16T12:16:30.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:29 smithi173 ceph-mon[36632]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:16:30.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:29 smithi098 ceph-mon[31633]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:16:30.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:29 smithi153 ceph-mon[36901]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T12:16:32.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:31 smithi173 ceph-mon[36632]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:32.103 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:31 smithi098 ceph-mon[31633]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:32.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:31 smithi153 ceph-mon[36901]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:34.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:33 smithi153 ceph-mon[36901]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:34.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:33 smithi173 ceph-mon[36632]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:34.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:33 smithi098 ceph-mon[31633]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:35.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T12:16:35.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:35.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:34 smithi173 ceph-mon[36632]: Deploying daemon osd.9 on smithi098 2024-03-16T12:16:35.120 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T12:16:35.120 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:35.120 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:34 smithi098 ceph-mon[31633]: Deploying daemon osd.9 on smithi098 2024-03-16T12:16:35.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T12:16:35.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:35.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:34 smithi153 ceph-mon[36901]: Deploying daemon osd.9 on smithi098 2024-03-16T12:16:35.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:35 smithi098 ceph-mon[31633]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:36.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:35 smithi173 ceph-mon[36632]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:36.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:35 smithi153 ceph-mon[36901]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:38.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:37 smithi173 ceph-mon[36632]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:37 smithi098 ceph-mon[31633]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:38.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:37 smithi153 ceph-mon[36901]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:39.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:39 smithi098 ceph-mon[31633]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:40.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:39 smithi173 ceph-mon[36632]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:39 smithi153 ceph-mon[36901]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:41 smithi098 ceph-mon[31633]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:42.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:41 smithi173 ceph-mon[36632]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:42.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:41 smithi153 ceph-mon[36901]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:43.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:43.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:43 smithi098 ceph-mon[31633]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:43.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:43.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:43 smithi173 ceph-mon[36632]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:43.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:43.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:43.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:43 smithi153 ceph-mon[36901]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:46.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:45 smithi098 ceph-mon[31633]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:46.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:45 smithi173 ceph-mon[36632]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:46.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:45 smithi153 ceph-mon[36901]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:47.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:47 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:47 smithi098 ceph-mon[31633]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:47 smithi098 ceph-mon[31633]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:47 smithi098 ceph-mon[31633]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:47.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:47 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:47 smithi173 ceph-mon[36632]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:47 smithi173 ceph-mon[36632]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:47 smithi173 ceph-mon[36632]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:47.424 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-03-16T12:16:47.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.440+0000 7f44ca000700 1 -- 172.21.15.98:0/3140634314 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f44cc062710 con 0x7f44b40a2910 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 msgr2=0x7f44b40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 0x7f44b40a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f44c00038c0 tx=0x7f44c0005c60 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 msgr2=0x7f44cc19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44b800e9f0 tx=0x7f44b800ed00 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 shutdown_connections 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f44b40a2910 0x7f44b40a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44cc108950 0x7f44cc19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44cc10a4f0 0x7f44cc1a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 --2- 172.21.15.98:0/3140634314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44cc109b50 0x7f44cc19f0b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 >> 172.21.15.98:0/3140634314 conn(0x7f44cc077920 msgr2=0x7f44cc10ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:16:47.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 shutdown_connections 2024-03-16T12:16:47.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:16:47.444+0000 7f44d4400700 1 -- 172.21.15.98:0/3140634314 wait complete. 2024-03-16T12:16:47.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:47 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:47.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:47 smithi153 ceph-mon[36901]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:47 smithi153 ceph-mon[36901]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T12:16:47.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:47 smithi153 ceph-mon[36901]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:48.196 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T12:16:48.202 INFO:tasks.cephadm:Deploying osd.10 on smithi153 with /dev/vg_nvme/lv_1... 2024-03-16T12:16:48.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:48.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:48 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:48.452 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:48 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:48.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:48 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:16:49.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:49 smithi098 ceph-mon[31633]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:16:49.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:49 smithi098 ceph-mon[31633]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T12:16:49.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:49 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:49 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:49 smithi098 ceph-mon[31633]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:49 smithi153 ceph-mon[36901]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:49 smithi153 ceph-mon[36901]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:49 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:49 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:49 smithi153 ceph-mon[36901]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:49.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:49 smithi173 ceph-mon[36632]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-16T12:16:49.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:49 smithi173 ceph-mon[36632]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T12:16:49.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:49 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:49 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:49.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:49 smithi173 ceph-mon[36632]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-16T12:16:50.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:50 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:16:50.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:50 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:16:50.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:50 smithi153 ceph-mon[36901]: osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] boot 2024-03-16T12:16:50.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:50 smithi153 ceph-mon[36901]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T12:16:50.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:50 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:50.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:50 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:16:50.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:50 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:16:50.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:50 smithi173 ceph-mon[36632]: osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] boot 2024-03-16T12:16:50.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:50 smithi173 ceph-mon[36632]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T12:16:50.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:50 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:50.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:50 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:16:50.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:50 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:16:50.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:50 smithi098 ceph-mon[31633]: osd.9 [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] boot 2024-03-16T12:16:50.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:50 smithi098 ceph-mon[31633]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T12:16:50.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:50 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123416 s, 850 MB/s 2024-03-16T12:16:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-16T12:16:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:51 smithi153 ceph-mon[36901]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T12:16:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:51 smithi153 ceph-mon[36901]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:51.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-03-16T12:16:51.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:51 smithi173 ceph-mon[36632]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T12:16:51.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:51 smithi173 ceph-mon[36632]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:51.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:51 smithi098 ceph-mon[31633]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T12:16:51.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:51 smithi098 ceph-mon[31633]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:51.523 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:16:53.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 -- 172.21.15.153:0/1815998067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 msgr2=0x7f89a0109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 --2- 172.21.15.153:0/1815998067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a0109640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8998009fc0 tx=0x7f899800b330 comp rx=0 tx=0).stop 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 -- 172.21.15.153:0/1815998067 shutdown_connections 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 --2- 172.21.15.153:0/1815998067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a0109640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 --2- 172.21.15.153:0/1815998067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01011f0 0x7f89a0101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 --2- 172.21.15.153:0/1815998067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 0x7f89a0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 -- 172.21.15.153:0/1815998067 >> 172.21.15.153:0/1815998067 conn(0x7f89a00fc790 msgr2=0x7f89a00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 -- 172.21.15.153:0/1815998067 shutdown_connections 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 -- 172.21.15.153:0/1815998067 wait complete. 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.262+0000 7f89a5709700 1 Processor -- start 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 -- start start 2024-03-16T12:16:53.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01011f0 0x7f89a006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 0x7f89a019d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010cdd0 con 0x7f89a01011f0 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010cc50 con 0x7f89a0102d90 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f89a5709700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010cf50 con 0x7f89a01023f0 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59184/0 (socket says 172.21.15.153:59184) 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 -- 172.21.15.153:0/30439358 learned_addr learned my addr 172.21.15.153:0/30439358 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899e7fc700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 0x7f89a019d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 -- 172.21.15.153:0/30439358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 msgr2=0x7f89a019d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 0x7f89a019d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 -- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01011f0 msgr2=0x7f89a006b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:16:53.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01011f0 0x7f89a006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.263+0000 7f899f7fe700 1 -- 172.21.15.153:0/30439358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a01a00e0 con 0x7f89a0102d90 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f899f7fe700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8998007c80 tx=0x7f8998007cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998021020 con 0x7f89a0102d90 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a01a0370 con 0x7f89a0102d90 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8998004e40 con 0x7f89a0102d90 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998005450 con 0x7f89a0102d90 2024-03-16T12:16:53.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.264+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a01a0820 con 0x7f89a0102d90 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.265+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a0068f90 con 0x7f89a0102d90 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.266+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8998024020 con 0x7f89a0102d90 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.266+0000 7f8987fff700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 0x7f89880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.267+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f89980cf690 con 0x7f89a0102d90 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.267+0000 7f899effd700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 0x7f89880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:16:53.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.268+0000 7f899effd700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 0x7f89880a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f89a0102250 tx=0x7f8990005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:16:53.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.271+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f899802b020 con 0x7f89a0102d90 2024-03-16T12:16:53.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:16:53.478+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f89a0062710 con 0x7f89880a2890 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:54.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:16:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:16:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:16:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:16:54.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:16:55.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:54 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi098 2024-03-16T12:16:55.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:54 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T12:16:55.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:54 smithi173 ceph-mon[36632]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:55.108 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:54 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi098 2024-03-16T12:16:55.108 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:54 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T12:16:55.109 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:54 smithi153 ceph-mon[36901]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:55.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:54 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi098 2024-03-16T12:16:55.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:54 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi098 to 4272M 2024-03-16T12:16:55.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:54 smithi098 ceph-mon[31633]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:16:56.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:55 smithi173 ceph-mon[36632]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:56.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:55 smithi098 ceph-mon[31633]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:56.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:55 smithi153 ceph-mon[36901]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:56.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/3828505629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:56.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:56 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:56.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:56 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]': finished 2024-03-16T12:16:56.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:56 smithi098 ceph-mon[31633]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T12:16:56.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:56 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:16:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:56 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/3828505629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:56 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:56 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]': finished 2024-03-16T12:16:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:56 smithi153 ceph-mon[36901]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T12:16:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:56 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:16:57.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:56 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/3828505629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:57.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:56 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]: dispatch 2024-03-16T12:16:57.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:56 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deab615a-9d50-4480-ac75-93033b168f40"}]': finished 2024-03-16T12:16:57.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:56 smithi173 ceph-mon[36632]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T12:16:57.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:56 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:16:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:57 smithi173 ceph-mon[36632]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:57 smithi173 ceph-mon[36632]: from='client.? 172.21.15.153:0/3670752369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:16:58.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:57 smithi098 ceph-mon[31633]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:58.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:57 smithi098 ceph-mon[31633]: from='client.? 172.21.15.153:0/3670752369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:16:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:57 smithi153 ceph-mon[36901]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:16:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:57 smithi153 ceph-mon[36901]: from='client.? 172.21.15.153:0/3670752369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:17:00.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:16:59 smithi173 ceph-mon[36632]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:00.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:16:59 smithi098 ceph-mon[31633]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:00.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:16:59 smithi153 ceph-mon[36901]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:02.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:01 smithi153 ceph-mon[36901]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:01 smithi173 ceph-mon[36632]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:02.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:01 smithi098 ceph-mon[31633]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:04.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:03 smithi173 ceph-mon[36632]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:04.112 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:03 smithi153 ceph-mon[36901]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:04.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:03 smithi098 ceph-mon[31633]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:05.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:04 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T12:17:05.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:04 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:05.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:04 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T12:17:05.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:04 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:05.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T12:17:05.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:04 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:05 smithi173 ceph-mon[36632]: Deploying daemon osd.10 on smithi153 2024-03-16T12:17:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:05 smithi173 ceph-mon[36632]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:06.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:05 smithi153 ceph-mon[36901]: Deploying daemon osd.10 on smithi153 2024-03-16T12:17:06.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:05 smithi153 ceph-mon[36901]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:05 smithi098 ceph-mon[31633]: Deploying daemon osd.10 on smithi153 2024-03-16T12:17:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:05 smithi098 ceph-mon[31633]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:08.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:07 smithi153 ceph-mon[36901]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:07 smithi098 ceph-mon[31633]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:08.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:07 smithi173 ceph-mon[36632]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:09.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:09 smithi153 ceph-mon[36901]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:10.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:09 smithi098 ceph-mon[31633]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:10.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:09 smithi173 ceph-mon[36632]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:11.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:11.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:11 smithi153 ceph-mon[36901]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:11.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:11.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:11 smithi173 ceph-mon[36632]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:11.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:11.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:11.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:11 smithi098 ceph-mon[31633]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:13.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:13 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.967+0000 7f8987fff700 1 -- 172.21.15.153:0/30439358 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f89a0062710 con 0x7f89880a2890 2024-03-16T12:17:13.951 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 10 on host 'smithi153' 2024-03-16T12:17:13.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 msgr2=0x7f89880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:13.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 0x7f89880a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f89a0102250 tx=0x7f8990005ca0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 msgr2=0x7f89a019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8998007c80 tx=0x7f8998007cb0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 shutdown_connections 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89880a2890 0x7f89880a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89a0102d90 0x7f89a019f810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89a01011f0 0x7f89a006b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 --2- 172.21.15.153:0/30439358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a01023f0 0x7f89a019d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 >> 172.21.15.153:0/30439358 conn(0x7f89a00fc790 msgr2=0x7f89a0105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 shutdown_connections 2024-03-16T12:17:13.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-16T12:17:13.970+0000 7f89a5709700 1 -- 172.21.15.153:0/30439358 wait complete. 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:14.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:13 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:14.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:13 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.664 DEBUG:teuthology.orchestra.run.smithi153:osd.10> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T12:17:14.666 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-03-16T12:17:14.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T12:17:14.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:14 smithi153 ceph-mon[36901]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:14.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:14 smithi153 ceph-mon[36901]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:14.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:14 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:14 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:14.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:14 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:14.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:17:15.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:14 smithi173 ceph-mon[36632]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:15.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:14 smithi173 ceph-mon[36632]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:15.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:14 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:15.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:14 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:15.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:14 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:15.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:14 smithi098 ceph-mon[31633]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:15.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:14 smithi098 ceph-mon[31633]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T12:17:15.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:14 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:15.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:14 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:15.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:14 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:15.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:15.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-mon[36901]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:17:15.888 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:17:15.812+0000 7f4422621700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T12:17:15.888 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:17:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:17:15.827+0000 7f441e414700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:15 smithi173 ceph-mon[36632]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: from='osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-16T12:17:16.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:15 smithi098 ceph-mon[31633]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130522 s, 803 MB/s 2024-03-16T12:17:16.977 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-16T12:17:16.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:16 smithi173 ceph-mon[36632]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T12:17:16.978 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:16 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:16.978 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:16 smithi173 ceph-mon[36632]: osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] boot 2024-03-16T12:17:16.978 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:16 smithi173 ceph-mon[36632]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T12:17:16.978 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:16 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:17.096 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:16 smithi153 ceph-mon[36901]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T12:17:17.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:16 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:17.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:16 smithi153 ceph-mon[36901]: osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] boot 2024-03-16T12:17:17.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:16 smithi153 ceph-mon[36901]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T12:17:17.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:16 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:17.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:16 smithi098 ceph-mon[31633]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T12:17:17.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:17.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:16 smithi098 ceph-mon[31633]: osd.10 [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] boot 2024-03-16T12:17:17.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:16 smithi098 ceph-mon[31633]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T12:17:17.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:16 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T12:17:17.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-03-16T12:17:18.006 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:17:18.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:17 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:17:18.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:17 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:17:18.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:17 smithi173 ceph-mon[36632]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:18.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:17 smithi173 ceph-mon[36632]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T12:17:18.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:17 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:17:18.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:17 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:17:18.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:17 smithi098 ceph-mon[31633]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:18.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:17 smithi098 ceph-mon[31633]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T12:17:18.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:17 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:17:18.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:17 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:17:18.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:17 smithi153 ceph-mon[36901]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-16T12:17:18.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:17 smithi153 ceph-mon[36901]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi153 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:18 smithi173 ceph-mon[36632]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi153 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:19.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:19.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:18 smithi098 ceph-mon[31633]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi153 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi153 to 4270M 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:19.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:19.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:18 smithi153 ceph-mon[36901]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T12:17:19.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- 172.21.15.173:0/120199747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc107ff0 msgr2=0x7f14bc10a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:19.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 --2- 172.21.15.173:0/120199747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc107ff0 0x7f14bc10a460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f14b400b0a0 tx=0x7f14b400b3b0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- 172.21.15.173:0/120199747 shutdown_connections 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 --2- 172.21.15.173:0/120199747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc107ff0 0x7f14bc10a460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 --2- 172.21.15.173:0/120199747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc102d90 0x7f14bc105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 --2- 172.21.15.173:0/120199747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc1056c0 0x7f14bc107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- 172.21.15.173:0/120199747 >> 172.21.15.173:0/120199747 conn(0x7f14bc0fc7c0 msgr2=0x7f14bc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- 172.21.15.173:0/120199747 shutdown_connections 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- 172.21.15.173:0/120199747 wait complete. 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 Processor -- start 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.693+0000 7f14c28a4700 1 -- start start 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc1056c0 0x7f14bc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc10df50 con 0x7f14bc1056c0 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc10ddd0 con 0x7f14bc107ff0 2024-03-16T12:17:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c28a4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc10e0d0 con 0x7f14bc102d90 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c0e41700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c0e41700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:47982/0 (socket says 172.21.15.173:47982) 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c0e41700 1 -- 172.21.15.173:0/1582324570 learned_addr learned my addr 172.21.15.173:0/1582324570 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:43402/0 (socket says 172.21.15.173:43402) 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bb7fe700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc1056c0 0x7f14bc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 -- 172.21.15.173:0/1582324570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 msgr2=0x7f14bc101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 -- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc1056c0 msgr2=0x7f14bc100d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc1056c0 0x7f14bc100d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 -- 172.21.15.173:0/1582324570 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc101b30 con 0x7f14bc102d90 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14c0e41700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:17:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.694+0000 7f14bbfff700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b450 tx=0x7f14b000b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:19.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.695+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0015020 con 0x7f14bc102d90 2024-03-16T12:17:19.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.695+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc1a26c0 con 0x7f14bc102d90 2024-03-16T12:17:19.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.695+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b0011d10 con 0x7f14bc102d90 2024-03-16T12:17:19.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.695+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0010e20 con 0x7f14bc102d90 2024-03-16T12:17:19.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.695+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc1a2bd0 con 0x7f14bc102d90 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.696+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14bc068f90 con 0x7f14bc102d90 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.697+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f14b0018020 con 0x7f14bc102d90 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.697+0000 7f14b97fa700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 0x7f14a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.698+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f14b00cdfe0 con 0x7f14bc102d90 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.698+0000 7f14bb7fe700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 0x7f14a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:19.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.699+0000 7f14bb7fe700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 0x7f14a40a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14ac005950 tx=0x7f14ac0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:19.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.701+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14b001f020 con 0x7f14bc102d90 2024-03-16T12:17:19.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:19.903+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f14bc062710 con 0x7f14a40a2890 2024-03-16T12:17:20.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:19 smithi173 ceph-mon[36632]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:20.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:19 smithi173 ceph-mon[36632]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T12:17:20.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:19 smithi098 ceph-mon[31633]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:20.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:19 smithi098 ceph-mon[31633]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T12:17:20.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:19 smithi153 ceph-mon[36901]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:20.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:19 smithi153 ceph-mon[36901]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T12:17:21.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:20 smithi173 ceph-mon[36632]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:21.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:17:21.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:17:21.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:21.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:20 smithi173 ceph-mon[36632]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T12:17:21.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:20 smithi098 ceph-mon[31633]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:21.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:17:21.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:17:21.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:21.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:20 smithi098 ceph-mon[31633]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T12:17:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:20 smithi153 ceph-mon[36901]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T12:17:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T12:17:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:20 smithi153 ceph-mon[36901]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T12:17:22.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:21 smithi173 ceph-mon[36632]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-16T12:17:22.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:21 smithi098 ceph-mon[31633]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-16T12:17:22.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:21 smithi153 ceph-mon[36901]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-16T12:17:23.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:22 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/3124137946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:22 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:22 smithi098 ceph-mon[31633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]': finished 2024-03-16T12:17:23.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:22 smithi098 ceph-mon[31633]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T12:17:23.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:23.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:22 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/3124137946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:22 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:22 smithi153 ceph-mon[36901]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]': finished 2024-03-16T12:17:23.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:22 smithi153 ceph-mon[36901]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T12:17:23.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:23.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:22 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/3124137946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:22 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]: dispatch 2024-03-16T12:17:23.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:22 smithi173 ceph-mon[36632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae"}]': finished 2024-03-16T12:17:23.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:22 smithi173 ceph-mon[36632]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T12:17:23.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:23 smithi098 ceph-mon[31633]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-16T12:17:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.173:0/3940451059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:17:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:23 smithi153 ceph-mon[36901]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-16T12:17:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.173:0/3940451059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:17:24.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:23 smithi173 ceph-mon[36632]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-16T12:17:24.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.173:0/3940451059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T12:17:26.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:25 smithi098 ceph-mon[31633]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:17:26.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:25 smithi153 ceph-mon[36901]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:17:26.308 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:25 smithi173 ceph-mon[36632]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T12:17:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:27 smithi098 ceph-mon[31633]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:17:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:27 smithi153 ceph-mon[36901]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:17:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:27 smithi173 ceph-mon[36632]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-16T12:17:30.036 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:29 smithi173 ceph-mon[36632]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:30.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:29 smithi098 ceph-mon[31633]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:30.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:29 smithi153 ceph-mon[36901]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:30.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T12:17:30.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:31.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T12:17:31.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:31.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T12:17:31.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:32.023 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:31 smithi173 ceph-mon[36632]: Deploying daemon osd.11 on smithi173 2024-03-16T12:17:32.023 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:31 smithi173 ceph-mon[36632]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:31 smithi098 ceph-mon[31633]: Deploying daemon osd.11 on smithi173 2024-03-16T12:17:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:31 smithi098 ceph-mon[31633]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:32.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:31 smithi153 ceph-mon[36901]: Deploying daemon osd.11 on smithi173 2024-03-16T12:17:32.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:31 smithi153 ceph-mon[36901]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:34.215 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:33 smithi173 ceph-mon[36632]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:34.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:33 smithi098 ceph-mon[31633]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:34.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:33 smithi153 ceph-mon[36901]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:36.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:35 smithi173 ceph-mon[36632]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:36.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:36.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:36.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:35 smithi098 ceph-mon[31633]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:36.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:35 smithi153 ceph-mon[36901]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:36.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:36.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:36.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:38.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:37 smithi173 ceph-mon[36632]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:37 smithi098 ceph-mon[31633]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:37 smithi153 ceph-mon[36901]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:39.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.701+0000 7f14b97fa700 1 -- 172.21.15.173:0/1582324570 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f14bc062710 con 0x7f14a40a2890 2024-03-16T12:17:39.685 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.703+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 msgr2=0x7f14a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.703+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 0x7f14a40a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14ac005950 tx=0x7f14ac0058e0 comp rx=0 tx=0).stop 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.703+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 msgr2=0x7f14bc1026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.703+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b450 tx=0x7f14b000b760 comp rx=0 tx=0).stop 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 shutdown_connections 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14a40a2890 0x7f14a40a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14bc107ff0 0x7f14bc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14bc102d90 0x7f14bc1026f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 --2- 172.21.15.173:0/1582324570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14bc1056c0 0x7f14bc100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:39.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 >> 172.21.15.173:0/1582324570 conn(0x7f14bc0fc7c0 msgr2=0x7f14bc103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:39.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 shutdown_connections 2024-03-16T12:17:39.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:39.704+0000 7f14c28a4700 1 -- 172.21.15.173:0/1582324570 wait complete. 2024-03-16T12:17:40.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:39 smithi098 ceph-mon[31633]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:39 smithi153 ceph-mon[36901]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:39 smithi173 ceph-mon[36632]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T12:17:40.363 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T12:17:40.365 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T12:17:40.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T12:17:40.814 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:17:41.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T12:17:41.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:41.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T12:17:41.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:41.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:41.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:41 smithi173 ceph-mon[36632]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:41.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:41 smithi098 ceph-mon[31633]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:42.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T12:17:42.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: from='osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:42.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T12:17:42.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:42.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-16T12:17:42.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:41 smithi153 ceph-mon[36901]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:42.639 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:17:42.349+0000 7f1421874700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T12:17:42.639 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:17:42.366+0000 7f1418e5e700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T12:17:42.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: purged_snaps scrub starts 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: purged_snaps scrub ok 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:42.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:42 smithi098 ceph-mon[31633]: from='osd.11 ' entity='osd.11' 2024-03-16T12:17:43.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: purged_snaps scrub starts 2024-03-16T12:17:43.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: purged_snaps scrub ok 2024-03-16T12:17:43.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:17:43.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T12:17:43.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:42 smithi153 ceph-mon[36901]: from='osd.11 ' entity='osd.11' 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: purged_snaps scrub starts 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: purged_snaps scrub ok 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:42 smithi173 ceph-mon[36632]: from='osd.11 ' entity='osd.11' 2024-03-16T12:17:43.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 -- 172.21.15.98:0/2366384657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020104550 msgr2=0x7f7020104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:43.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 --2- 172.21.15.98:0/2366384657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020104550 0x7f7020104970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70100098f0 tx=0x7f7010009c00 comp rx=0 tx=0).stop 2024-03-16T12:17:43.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 -- 172.21.15.98:0/2366384657 shutdown_connections 2024-03-16T12:17:43.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 --2- 172.21.15.98:0/2366384657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70201060f0 0x7f702010c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 --2- 172.21.15.98:0/2366384657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020104550 0x7f7020104970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 --2- 172.21.15.98:0/2366384657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7020105750 0x7f7020105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 -- 172.21.15.98:0/2366384657 >> 172.21.15.98:0/2366384657 conn(0x7f70200ffaf0 msgr2=0x7f7020101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 -- 172.21.15.98:0/2366384657 shutdown_connections 2024-03-16T12:17:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.178+0000 7f7025c00700 1 -- 172.21.15.98:0/2366384657 wait complete. 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.179+0000 7f7025c00700 1 Processor -- start 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 -- start start 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020105750 0x7f702019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70201060f0 0x7f70201a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7020110170 con 0x7f70201060f0 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f702010fff0 con 0x7f7020104550 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f7025c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70201102f0 con 0x7f7020105750 2024-03-16T12:17:43.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f701ea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f701ea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45250/0 (socket says 172.21.15.98:45250) 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.180+0000 7f701ea00700 1 -- 172.21.15.98:0/3460441590 learned_addr learned my addr 172.21.15.98:0/3460441590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701e000700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020105750 0x7f702019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701f400700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70201060f0 0x7f70201a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 -- 172.21.15.98:0/3460441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020105750 msgr2=0x7f702019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020105750 0x7f702019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 -- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70201060f0 msgr2=0x7f70201a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70201060f0 0x7f70201a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 -- 172.21.15.98:0/3460441590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70201a23e0 con 0x7f7020104550 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f701ea00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7010009230 tx=0x7f70100052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.181+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701000b560 con 0x7f7020104550 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.182+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70201a2670 con 0x7f7020104550 2024-03-16T12:17:43.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.182+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f701000bbe0 con 0x7f7020104550 2024-03-16T12:17:43.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.182+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010017400 con 0x7f7020104550 2024-03-16T12:17:43.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.182+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70201a2b20 con 0x7f7020104550 2024-03-16T12:17:43.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.183+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7020068f90 con 0x7f7020104550 2024-03-16T12:17:43.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.185+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7010003680 con 0x7f7020104550 2024-03-16T12:17:43.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.186+0000 7f7017400700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 0x7f70040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:43.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.186+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f70100cf660 con 0x7f7020104550 2024-03-16T12:17:43.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.186+0000 7f701e000700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 0x7f70040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:43.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.187+0000 7f701e000700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 0x7f70040a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f700c009750 tx=0x7f700c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:43.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.188+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7010013030 con 0x7f7020104550 2024-03-16T12:17:43.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.412+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f702002db70 con 0x7f7020104550 2024-03-16T12:17:43.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.413+0000 7f7017400700 1 -- 172.21.15.98:0/3460441590 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f701009ccb0 con 0x7f7020104550 2024-03-16T12:17:43.397 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:17:43.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.417+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 msgr2=0x7f70040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:43.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.417+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 0x7f70040a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f700c009750 tx=0x7f700c009310 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.417+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 msgr2=0x7f702019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.417+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7010009230 tx=0x7f70100052f0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 shutdown_connections 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70040a2890 0x7f70040a4d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7020104550 0x7f702019a780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70201060f0 0x7f70201a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 --2- 172.21.15.98:0/3460441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7020105750 0x7f702019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 >> 172.21.15.98:0/3460441590 conn(0x7f70200ffaf0 msgr2=0x7f7020108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:43.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 shutdown_connections 2024-03-16T12:17:43.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:43.418+0000 7f7025c00700 1 -- 172.21.15.98:0/3460441590 wait complete. 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] boot 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3460441590' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T12:17:43.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:43.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] boot 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3460441590' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:44.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: osd.11 [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] boot 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3460441590' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:44.207 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710591463,"num_in_osds":12,"osd_in_since":1710591442,"num_remapped_pgs":0} 2024-03-16T12:17:44.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T12:17:44.619 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:17:44.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:44 smithi098 ceph-mon[31633]: Detected new or changed devices on smithi173 2024-03-16T12:17:44.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:44 smithi098 ceph-mon[31633]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-16T12:17:44.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:44 smithi098 ceph-mon[31633]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T12:17:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:44 smithi153 ceph-mon[36901]: Detected new or changed devices on smithi173 2024-03-16T12:17:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:44 smithi153 ceph-mon[36901]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-16T12:17:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:44 smithi153 ceph-mon[36901]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T12:17:45.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:44 smithi173 ceph-mon[36632]: Detected new or changed devices on smithi173 2024-03-16T12:17:45.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:44 smithi173 ceph-mon[36632]: Adjusting osd_memory_target on smithi173 to 4270M 2024-03-16T12:17:45.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:44 smithi173 ceph-mon[36632]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T12:17:45.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:45 smithi098 ceph-mon[31633]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:46.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:45 smithi153 ceph-mon[36901]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:46.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:45 smithi173 ceph-mon[36632]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 -- 172.21.15.98:0/7833063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0730107650 msgr2=0x7f073010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 --2- 172.21.15.98:0/7833063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0730107650 0x7f073010f1d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f072800b0d0 tx=0x7f072800b3e0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 -- 172.21.15.98:0/7833063 shutdown_connections 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 --2- 172.21.15.98:0/7833063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073006ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 --2- 172.21.15.98:0/7833063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0730107650 0x7f073010f1d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 --2- 172.21.15.98:0/7833063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073006b420 0x7f0730106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 -- 172.21.15.98:0/7833063 >> 172.21.15.98:0/7833063 conn(0x7f0730077220 msgr2=0x7f0730077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 -- 172.21.15.98:0/7833063 shutdown_connections 2024-03-16T12:17:46.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.734+0000 7f0736e00700 1 -- 172.21.15.98:0/7833063 wait complete. 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.735+0000 7f0736e00700 1 Processor -- start 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.735+0000 7f0736e00700 1 -- start start 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 0x7f0730198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0730112660 con 0x7f073006b420 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07301124e0 con 0x7f073006aa70 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0736e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f07301127e0 con 0x7f0730107650 2024-03-16T12:17:46.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0727e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0727e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58876/0 (socket says 172.21.15.98:58876) 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0727e00700 1 -- 172.21.15.98:0/577715641 learned_addr learned my addr 172.21.15.98:0/577715641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f072fe00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.736+0000 7f0734e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 0x7f0730198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 -- 172.21.15.98:0/577715641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 msgr2=0x7f073019e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073019e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 -- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 msgr2=0x7f073019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 -- 172.21.15.98:0/577715641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07301990c0 con 0x7f0730107650 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0727e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f072fe00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:17:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0734e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 0x7f0730198960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f072800d010 tx=0x7f0728009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728003a00 con 0x7f0730107650 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0728022070 con 0x7f0730107650 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072801b6c0 con 0x7f0730107650 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0730199350 con 0x7f0730107650 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.737+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0730101200 con 0x7f0730107650 2024-03-16T12:17:46.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.738+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0730068f90 con 0x7f0730107650 2024-03-16T12:17:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.741+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0728003ba0 con 0x7f0730107650 2024-03-16T12:17:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.742+0000 7f072d600700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 0x7f07140a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.742+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f072800f020 con 0x7f0730107650 2024-03-16T12:17:46.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.742+0000 7f072fe00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 0x7f07140a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:46.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.743+0000 7f072fe00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 0x7f07140a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0730199aa0 tx=0x7f0720012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:46.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.744+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f072809cad0 con 0x7f0730107650 2024-03-16T12:17:46.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.970+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f073002db70 con 0x7f0730107650 2024-03-16T12:17:46.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.971+0000 7f072d600700 1 -- 172.21.15.98:0/577715641 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7f072809c3d0 con 0x7f0730107650 2024-03-16T12:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:17:44.481936+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:17:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 msgr2=0x7f07140a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 0x7f07140a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0730199aa0 tx=0x7f0720012040 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 msgr2=0x7f0730198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 0x7f0730198960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f072800d010 tx=0x7f0728009810 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 shutdown_connections 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f07140a2910 0x7f07140a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f073006aa70 0x7f073019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0730107650 0x7f0730198960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 --2- 172.21.15.98:0/577715641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f073006b420 0x7f073019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 >> 172.21.15.98:0/577715641 conn(0x7f0730077220 msgr2=0x7f07300ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 shutdown_connections 2024-03-16T12:17:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:46.975+0000 7f0736e00700 1 -- 172.21.15.98:0/577715641 wait complete. 2024-03-16T12:17:47.958 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T12:17:47.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T12:17:48.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:47 smithi098 ceph-mon[31633]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:48.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:47 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/577715641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:48.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:47 smithi173 ceph-mon[36632]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:48.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:47 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/577715641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:48.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:47 smithi153 ceph-mon[36901]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:48.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:47 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/577715641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:17:49.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:49 smithi098 ceph-mon[31633]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:50.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:49 smithi173 ceph-mon[36632]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:50.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:49 smithi153 ceph-mon[36901]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 -- 172.21.15.98:0/1557843386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb010ed00 msgr2=0x7f5fb01110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/1557843386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb010ed00 0x7f5fb01110f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800b0a0 tx=0x7f5fa800b3b0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 -- 172.21.15.98:0/1557843386 shutdown_connections 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/1557843386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb010ed00 0x7f5fb01110f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/1557843386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb00748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/1557843386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb0074e20 0x7f5fb010e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 -- 172.21.15.98:0/1557843386 >> 172.21.15.98:0/1557843386 conn(0x7f5fb0079390 msgr2=0x7f5fb00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 -- 172.21.15.98:0/1557843386 shutdown_connections 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.592+0000 7f5fb6c00700 1 -- 172.21.15.98:0/1557843386 wait complete. 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.593+0000 7f5fb6c00700 1 Processor -- start 2024-03-16T12:17:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 -- start start 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb01a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 0x7f5fb01a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 0x7f5fb019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01147e0 con 0x7f5fb0074e20 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb0114660 con 0x7f5fb010ed00 2024-03-16T12:17:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb0114960 con 0x7f5fb0074500 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 0x7f5fb01a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fafe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb01a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 0x7f5fb01a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45158/0 (socket says 172.21.15.98:45158) 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fa7400700 1 -- 172.21.15.98:0/732540068 learned_addr learned my addr 172.21.15.98:0/732540068 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.594+0000 7f5fb4c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 0x7f5fb019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 -- 172.21.15.98:0/732540068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 msgr2=0x7f5fb019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 0x7f5fb019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 -- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 msgr2=0x7f5fb01a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 0x7f5fb01a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 -- 172.21.15.98:0/732540068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb019b1c0 con 0x7f5fb0074500 2024-03-16T12:17:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fb4c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 0x7f5fb019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fafe00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb01a0850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000e9f0 tx=0x7f5fa000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000cbd0 con 0x7f5fb0074500 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb019b4b0 con 0x7f5fb0074500 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fa000cd70 con 0x7f5fb0074500 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.595+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0018710 con 0x7f5fb0074500 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.596+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb01a7720 con 0x7f5fb0074500 2024-03-16T12:17:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.597+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb0068f90 con 0x7f5fb0074500 2024-03-16T12:17:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.599+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5fa0010430 con 0x7f5fb0074500 2024-03-16T12:17:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.600+0000 7f5fad600700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 0x7f5f940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.600+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5fa00cec80 con 0x7f5fb0074500 2024-03-16T12:17:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.600+0000 7f5fa7400700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 0x7f5f940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.601+0000 7f5fa7400700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 0x7f5f940a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009680 tx=0x7f5f98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.602+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fa009c120 con 0x7f5fb0074500 2024-03-16T12:17:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.826+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fb01a7980 con 0x7f5fb0074500 2024-03-16T12:17:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.826+0000 7f5fad600700 1 -- 172.21.15.98:0/732540068 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f5fa009ba20 con 0x7f5fb0074500 2024-03-16T12:17:50.810 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.829+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 msgr2=0x7f5f940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.829+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 0x7f5f940a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009680 tx=0x7f5f98009260 comp rx=0 tx=0).stop 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.829+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 msgr2=0x7f5fb01a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.829+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb01a0850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000e9f0 tx=0x7f5fa000ed00 comp rx=0 tx=0).stop 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 shutdown_connections 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5f940a2910 0x7f5f940a4dd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fb010ed00 0x7f5fb019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb0074500 0x7f5fb01a0850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 --2- 172.21.15.98:0/732540068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb0074e20 0x7f5fb01a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 >> 172.21.15.98:0/732540068 conn(0x7f5fb0079390 msgr2=0x7f5fb00ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 shutdown_connections 2024-03-16T12:17:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:50.830+0000 7f5fb6c00700 1 -- 172.21.15.98:0/732540068 wait complete. 2024-03-16T12:17:51.668 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.b on smithi153 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.e on smithi153 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.h on smithi153 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.k on smithi153 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.c on smithi173 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.f on smithi173 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.i on smithi173 2024-03-16T12:17:51.669 INFO:tasks.cephadm:Adding mds.l on smithi173 2024-03-16T12:17:51.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l' 2024-03-16T12:17:51.922 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:17:51.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:51 smithi098 ceph-mon[31633]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:51.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:51 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/732540068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:52.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:51 smithi173 ceph-mon[36632]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:52.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:51 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/732540068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:52.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:51 smithi153 ceph-mon[36901]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:52.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:51 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/732540068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 -- 172.21.15.173:0/188134288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102400 msgr2=0x7f0154102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 --2- 172.21.15.173:0/188134288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102400 0x7f0154102860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0148009990 tx=0x7f0148009ca0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 -- 172.21.15.173:0/188134288 shutdown_connections 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 --2- 172.21.15.173:0/188134288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154102da0 0x7f0154109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 --2- 172.21.15.173:0/188134288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0154101200 0x7f0154101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 --2- 172.21.15.173:0/188134288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102400 0x7f0154102860 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 -- 172.21.15.173:0/188134288 >> 172.21.15.173:0/188134288 conn(0x7f01540fc7a0 msgr2=0x7f01540febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 -- 172.21.15.173:0/188134288 shutdown_connections 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.584+0000 7f0159548700 1 -- 172.21.15.173:0/188134288 wait complete. 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 Processor -- start 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 -- start start 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154101200 0x7f0154105140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0154102400 0x7f01541037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f015410cee0 con 0x7f0154102400 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f015410cd60 con 0x7f0154101200 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.585+0000 7f0159548700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f015410d060 con 0x7f0154102da0 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:45056/0 (socket says 172.21.15.173:45056) 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 -- 172.21.15.173:0/3697319731 learned_addr learned my addr 172.21.15.173:0/3697319731 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f0152ffd700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154101200 0x7f0154105140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 -- 172.21.15.173:0/3697319731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154101200 msgr2=0x7f0154105140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154101200 0x7f0154105140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 -- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0154102400 msgr2=0x7f01541037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0154102400 0x7f01541037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 -- 172.21.15.173:0/3697319731 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01541045f0 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f01537fe700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f014c00b970 tx=0x7f014c00bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c019070 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01541a2840 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c014440 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01541a2e20 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.586+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0075d0 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.587+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0154068f90 con 0x7f0154102da0 2024-03-16T12:17:53.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.588+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f014c01c070 con 0x7f0154102da0 2024-03-16T12:17:53.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.589+0000 7f013bfff700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 0x7f013c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:53.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.589+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f014c0cdee0 con 0x7f0154102da0 2024-03-16T12:17:53.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.589+0000 7f0152ffd700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 0x7f013c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:53.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.590+0000 7f0152ffd700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 0x7f013c0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0144006fd0 tx=0x7f0144008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:53.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.593+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f014c09b420 con 0x7f0154102da0 2024-03-16T12:17:53.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.797+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}) -- 0x7f015406a790 con 0x7f013c0a2890 2024-03-16T12:17:53.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.815+0000 7f013bfff700 1 -- 172.21.15.173:0/3697319731 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f015406a790 con 0x7f013c0a2890 2024-03-16T12:17:53.798 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mds.all update... 2024-03-16T12:17:53.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 msgr2=0x7f013c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 0x7f013c0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0144006fd0 tx=0x7f0144008040 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 msgr2=0x7f0154103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f014c00b970 tx=0x7f014c00bd30 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 shutdown_connections 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f013c0a2890 0x7f013c0a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0154101200 0x7f0154105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0154102da0 0x7f0154103d20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 --2- 172.21.15.173:0/3697319731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0154102400 0x7f01541037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 >> 172.21.15.173:0/3697319731 conn(0x7f01540fc7a0 msgr2=0x7f0154069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 shutdown_connections 2024-03-16T12:17:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-16T12:17:53.817+0000 7f0159548700 1 -- 172.21.15.173:0/3697319731 wait complete. 2024-03-16T12:17:53.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:53 smithi173 ceph-mon[36632]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:53.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:53 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:54.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:53 smithi098 ceph-mon[31633]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:54.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:53 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:54.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:53 smithi153 ceph-mon[36901]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:54.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:53 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:54.505 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T12:17:54.510 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T12:17:54.556 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T12:17:54.601 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T12:17:54.646 DEBUG:teuthology.orchestra.run.smithi153:mds.b> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T12:17:54.649 DEBUG:teuthology.orchestra.run.smithi153:mds.e> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T12:17:54.691 DEBUG:teuthology.orchestra.run.smithi153:mds.h> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T12:17:54.734 DEBUG:teuthology.orchestra.run.smithi153:mds.k> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T12:17:54.777 DEBUG:teuthology.orchestra.run.smithi173:mds.c> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T12:17:54.780 DEBUG:teuthology.orchestra.run.smithi173:mds.f> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T12:17:54.824 DEBUG:teuthology.orchestra.run.smithi173:mds.i> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T12:17:54.868 DEBUG:teuthology.orchestra.run.smithi173:mds.l> sudo journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T12:17:54.910 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T12:17:54.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:17:54.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:54 smithi098 ceph-mon[31633]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:54.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:54 smithi098 ceph-mon[31633]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-03-16T12:17:54.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:54 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:55.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:54 smithi153 ceph-mon[36901]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:55.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:54 smithi153 ceph-mon[36901]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-03-16T12:17:55.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:54 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:55.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:54 smithi173 ceph-mon[36632]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:17:55.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:54 smithi173 ceph-mon[36632]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-03-16T12:17:55.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:54 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 -- 172.21.15.98:0/3830893102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74741055b0 msgr2=0x7f747410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 --2- 172.21.15.98:0/3830893102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74741055b0 0x7f747410be40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f746c009ca0 tx=0x7f746c02c100 comp rx=0 tx=0).stop 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 -- 172.21.15.98:0/3830893102 shutdown_connections 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 --2- 172.21.15.98:0/3830893102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74741055b0 0x7f747410be40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 --2- 172.21.15.98:0/3830893102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 0x7f7474105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 --2- 172.21.15.98:0/3830893102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474103a00 0x7f7474103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 -- 172.21.15.98:0/3830893102 >> 172.21.15.98:0/3830893102 conn(0x7f74740fef90 msgr2=0x7f74741013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 -- 172.21.15.98:0/3830893102 shutdown_connections 2024-03-16T12:17:55.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.167+0000 7f747b000640 1 -- 172.21.15.98:0/3830893102 wait complete. 2024-03-16T12:17:55.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 Processor -- start 2024-03-16T12:17:55.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 -- start start 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 0x7f747419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74741055b0 0x7f74741a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f747410f660 con 0x7f7474103a00 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f747410f4e0 con 0x7f74741055b0 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.169+0000 7f747b000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f747410f7e0 con 0x7f7474104bf0 2024-03-16T12:17:55.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45172/0 (socket says 172.21.15.98:45172) 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 -- 172.21.15.98:0/1211559654 learned_addr learned my addr 172.21.15.98:0/1211559654 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473400640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 0x7f747419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7479000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74741055b0 0x7f74741a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 -- 172.21.15.98:0/1211559654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 msgr2=0x7f747419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 0x7f747419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 -- 172.21.15.98:0/1211559654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74741055b0 msgr2=0x7f74741a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74741055b0 0x7f74741a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 -- 172.21.15.98:0/1211559654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74741a1870 con 0x7f7474103a00 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7473e00640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f746400e9c0 tx=0x7f746400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.170+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746400cde0 con 0x7f7474103a00 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.171+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7464010910 con 0x7f7474103a00 2024-03-16T12:17:55.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.171+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464018a00 con 0x7f7474103a00 2024-03-16T12:17:55.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.171+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74741a1b60 con 0x7f7474103a00 2024-03-16T12:17:55.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.171+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74741a2060 con 0x7f7474103a00 2024-03-16T12:17:55.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.172+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f747410acc0 con 0x7f7474103a00 2024-03-16T12:17:55.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.174+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f746401f020 con 0x7f7474103a00 2024-03-16T12:17:55.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.176+0000 7f7470c00640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.177+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f74640ce980 con 0x7f7474103a00 2024-03-16T12:17:55.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.177+0000 7f7473400640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.178+0000 7f7473400640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 0x7f74580a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f74600095f0 tx=0x7f7460005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.182+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f746409be20 con 0x7f7474103a00 2024-03-16T12:17:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.386+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7474062410 con 0x7f7474103a00 2024-03-16T12:17:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.387+0000 7f7470c00640 1 -- 172.21.15.98:0/1211559654 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7f746409b720 con 0x7f7474103a00 2024-03-16T12:17:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:17:55.371 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:17:44.481936+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:17:55.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 msgr2=0x7f74580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 0x7f74580a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f74600095f0 tx=0x7f7460005f90 comp rx=0 tx=0).stop 2024-03-16T12:17:55.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 msgr2=0x7f7474199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f746400e9c0 tx=0x7f746400ee80 comp rx=0 tx=0).stop 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 shutdown_connections 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74741055b0 0x7f74741a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7474104bf0 0x7f747419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 --2- 172.21.15.98:0/1211559654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7474103a00 0x7f7474199c60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 >> 172.21.15.98:0/1211559654 conn(0x7f74740fef90 msgr2=0x7f7474107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:55.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 shutdown_connections 2024-03-16T12:17:55.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.390+0000 7f747b000640 1 -- 172.21.15.98:0/1211559654 wait complete. 2024-03-16T12:17:55.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T12:17:55.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:17:55.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 -- 172.21.15.98:0/3633401873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7960108ff0 msgr2=0x7f7960109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 --2- 172.21.15.98:0/3633401873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7960108ff0 0x7f7960109470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f794c009860 tx=0x7f794c02bb70 comp rx=0 tx=0).stop 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 -- 172.21.15.98:0/3633401873 shutdown_connections 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 --2- 172.21.15.98:0/3633401873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79601099b0 0x7f7960110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 --2- 172.21.15.98:0/3633401873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7960108ff0 0x7f7960109470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 --2- 172.21.15.98:0/3633401873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f7960108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 -- 172.21.15.98:0/3633401873 >> 172.21.15.98:0/3633401873 conn(0x7f7960076da0 msgr2=0x7f79600791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 -- 172.21.15.98:0/3633401873 shutdown_connections 2024-03-16T12:17:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.624+0000 7f7964c00640 1 -- 172.21.15.98:0/3633401873 wait complete. 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.625+0000 7f7964c00640 1 Processor -- start 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.625+0000 7f7964c00640 1 -- start start 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7960108ff0 0x7f796019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79601099b0 0x7f79601a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7960113a60 con 0x7f7960108ff0 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79601138e0 con 0x7f79601099b0 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f7964c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7960113be0 con 0x7f7960107e00 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f795de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f795de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:54180/0 (socket says 172.21.15.98:54180) 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f795d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7960108ff0 0x7f796019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f795de00640 1 -- 172.21.15.98:0/213607466 learned_addr learned my addr 172.21.15.98:0/213607466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.626+0000 7f795e800640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79601099b0 0x7f79601a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 -- 172.21.15.98:0/213607466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79601099b0 msgr2=0x7f79601a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79601099b0 0x7f79601a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 -- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7960108ff0 msgr2=0x7f796019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7960108ff0 0x7f796019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 -- 172.21.15.98:0/213607466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79601a5d20 con 0x7f7960107e00 2024-03-16T12:17:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f795de00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f795000e9a0 tx=0x7f795000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795000cd80 con 0x7f7960107e00 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950010910 con 0x7f7960107e00 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79601a5fb0 con 0x7f7960107e00 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.627+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950018a40 con 0x7f7960107e00 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.628+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79601a6470 con 0x7f7960107e00 2024-03-16T12:17:55.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.629+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796010f120 con 0x7f7960107e00 2024-03-16T12:17:55.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.631+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7950010ac0 con 0x7f7960107e00 2024-03-16T12:17:55.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.632+0000 7f7956a00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 0x7f79440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:55.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.632+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79500ce8b0 con 0x7f7960107e00 2024-03-16T12:17:55.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.632+0000 7f795d400640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 0x7f79440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:55.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.633+0000 7f795d400640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 0x7f79440a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f794c02c180 tx=0x7f794c034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:55.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.636+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7950053740 con 0x7f7960107e00 2024-03-16T12:17:55.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.833+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f796002d460 con 0x7f7960107e00 2024-03-16T12:17:55.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.839+0000 7f7956a00640 1 -- 172.21.15.98:0/213607466 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f795009bdf0 con 0x7f7960107e00 2024-03-16T12:17:55.822 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.841+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 msgr2=0x7f79440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.841+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 0x7f79440a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f794c02c180 tx=0x7f794c034ad0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.841+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 msgr2=0x7f796019e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.841+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f795000e9a0 tx=0x7f795000ee60 comp rx=0 tx=0).stop 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 shutdown_connections 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79440a2800 0x7f79440a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79601099b0 0x7f79601a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7960108ff0 0x7f796019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 --2- 172.21.15.98:0/213607466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7960107e00 0x7f796019e100 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:55.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 >> 172.21.15.98:0/213607466 conn(0x7f7960076da0 msgr2=0x7f796010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:55.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 shutdown_connections 2024-03-16T12:17:55.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:55.842+0000 7f7964c00640 1 -- 172.21.15.98:0/213607466 wait complete. 2024-03-16T12:17:55.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:17:55.873 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: Deploying daemon mds.l on smithi173 2024-03-16T12:17:55.874 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:55 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1211559654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.071+0000 7f8f6e200640 1 -- 172.21.15.98:0/661181824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68104bf0 msgr2=0x7f8f68105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.071+0000 7f8f6e200640 1 --2- 172.21.15.98:0/661181824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68104bf0 0x7f8f68105070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009900 tx=0x7f8f5002bc30 comp rx=0 tx=0).stop 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 -- 172.21.15.98:0/661181824 shutdown_connections 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 --2- 172.21.15.98:0/661181824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681055b0 0x7f8f6810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 --2- 172.21.15.98:0/661181824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68104bf0 0x7f8f68105070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 --2- 172.21.15.98:0/661181824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68103a00 0x7f8f68103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 -- 172.21.15.98:0/661181824 >> 172.21.15.98:0/661181824 conn(0x7f8f680fef90 msgr2=0x7f8f681013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 -- 172.21.15.98:0/661181824 shutdown_connections 2024-03-16T12:17:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.072+0000 7f8f6e200640 1 -- 172.21.15.98:0/661181824 wait complete. 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 Processor -- start 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 -- start start 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68103a00 0x7f8f68199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 0x7f8f6819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f6b0 con 0x7f8f681055b0 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f530 con 0x7f8f68104bf0 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f6e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f830 con 0x7f8f68103a00 2024-03-16T12:17:56.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f5fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 0x7f8f6819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45192/0 (socket says 172.21.15.98:45192) 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f67e00640 1 -- 172.21.15.98:0/3525758223 learned_addr learned my addr 172.21.15.98:0/3525758223 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.073+0000 7f8f67400640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68103a00 0x7f8f68199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 -- 172.21.15.98:0/3525758223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68103a00 msgr2=0x7f8f68199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68103a00 0x7f8f68199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 -- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 msgr2=0x7f8f681a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 -- 172.21.15.98:0/3525758223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f681a18d0 con 0x7f8f68104bf0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f67e00640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f5fe00640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 0x7f8f6819a1d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f50036040 tx=0x7f8f5002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50043020 con 0x7f8f68104bf0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f500054e0 con 0x7f8f68104bf0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f681a1b60 con 0x7f8f68104bf0 2024-03-16T12:17:56.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.074+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50003c30 con 0x7f8f68104bf0 2024-03-16T12:17:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.075+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f681a2080 con 0x7f8f68104bf0 2024-03-16T12:17:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.076+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f6810ac50 con 0x7f8f68104bf0 2024-03-16T12:17:56.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.077+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8f50046020 con 0x7f8f68104bf0 2024-03-16T12:17:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.079+0000 7f8f64c00640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 0x7f8f4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.079+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8f500f0bf0 con 0x7f8f68104bf0 2024-03-16T12:17:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.079+0000 7f8f67400640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 0x7f8f4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.080+0000 7f8f67400640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 0x7f8f4c0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f6819ab60 tx=0x7f8f58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.081+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f5004d020 con 0x7f8f68104bf0 2024-03-16T12:17:56.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:56.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: Deploying daemon mds.l on smithi173 2024-03-16T12:17:56.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:55 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1211559654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:56.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: Deploying daemon mds.l on smithi173 2024-03-16T12:17:56.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:55 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1211559654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:17:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.305+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f6802d1b0 con 0x7f8f68104bf0 2024-03-16T12:17:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.306+0000 7f8f64c00640 1 -- 172.21.15.98:0/3525758223 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f8f500be090 con 0x7f8f68104bf0 2024-03-16T12:17:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:17:56.290 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T12:17:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-03-16T12:17:56.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 msgr2=0x7f8f4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:56.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 0x7f8f4c0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f6819ab60 tx=0x7f8f58005f90 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 msgr2=0x7f8f6819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 0x7f8f6819a1d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f50036040 tx=0x7f8f5002c650 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 shutdown_connections 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8f4c0a2800 0x7f8f4c0a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681055b0 0x7f8f681a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f68104bf0 0x7f8f6819a1d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 --2- 172.21.15.98:0/3525758223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68103a00 0x7f8f68199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 >> 172.21.15.98:0/3525758223 conn(0x7f8f680fef90 msgr2=0x7f8f68107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 shutdown_connections 2024-03-16T12:17:56.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:56.310+0000 7f8f6e200640 1 -- 172.21.15.98:0/3525758223 wait complete. 2024-03-16T12:17:56.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:56 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/213607466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:56.949 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:56 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3525758223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:57.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/213607466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:57.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:56 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3525758223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:57.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:56 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/213607466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:17:57.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:56 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3525758223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:57.321 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 systemd[1]: Starting Ceph mds.l for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:17:57.633 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 podman[66573]: 2024-03-16 12:17:57.266386163 +0000 UTC m=+0.017461235 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:17:57.633 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 podman[66573]: 2024-03-16 12:17:57.389793647 +0000 UTC m=+0.140868718 container create 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:17:57.633 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 podman[66573]: 2024-03-16 12:17:57.650888977 +0000 UTC m=+0.401964055 container init 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:17:58.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:58.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6832/307291521,v1:172.21.15.173:6833/307291521] up:boot 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: fsmap 1 up:standby 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 podman[66573]: 2024-03-16 12:17:57.653727939 +0000 UTC m=+0.404803019 container start 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-mds[66592]: -- 172.21.15.173:0/38370395 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55582f48ba40 con 0x55582f45ec00 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l[66588]: starting mds.l at 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 bash[66573]: 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be 2024-03-16T12:17:58.072 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:17:57 smithi173 systemd[1]: Started Ceph mds.l for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:17:58.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:58.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6832/307291521,v1:172.21.15.173:6833/307291521] up:boot 2024-03-16T12:17:58.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: fsmap 1 up:standby 2024-03-16T12:17:58.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T12:17:58.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:57 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6832/307291521,v1:172.21.15.173:6833/307291521] up:boot 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: fsmap 1 up:standby 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:57 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:58.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.551+0000 7fbebb600640 1 -- 172.21.15.98:0/1843133267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41033a0 msgr2=0x7fbeb4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.551+0000 7fbebb600640 1 --2- 172.21.15.98:0/1843133267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41033a0 0x7fbeb4109c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009be0 tx=0x7fbeac02bef0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 -- 172.21.15.98:0/1843133267 shutdown_connections 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 --2- 172.21.15.98:0/1843133267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41033a0 0x7fbeb4109c30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 --2- 172.21.15.98:0/1843133267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 0x7fbeb4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 --2- 172.21.15.98:0/1843133267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41017f0 0x7fbeb4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 -- 172.21.15.98:0/1843133267 >> 172.21.15.98:0/1843133267 conn(0x7fbeb40fcd80 msgr2=0x7fbeb40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 -- 172.21.15.98:0/1843133267 shutdown_connections 2024-03-16T12:17:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.552+0000 7fbebb600640 1 -- 172.21.15.98:0/1843133267 wait complete. 2024-03-16T12:17:58.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 Processor -- start 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 -- start start 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41017f0 0x7fbeb4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 0x7fbeb4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410d4a0 con 0x7fbeb41033a0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410d320 con 0x7fbeb41017f0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbebb600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410d620 con 0x7fbeb41029e0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbeb9600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbeb9600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57428/0 (socket says 172.21.15.98:57428) 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbeb9600640 1 -- 172.21.15.98:0/1082043966 learned_addr learned my addr 172.21.15.98:0/1082043966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.553+0000 7fbeabe00640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 0x7fbeb4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb8c00640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41017f0 0x7fbeb4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:58.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 -- 172.21.15.98:0/1082043966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 msgr2=0x7fbeb4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 0x7fbeb4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 -- 172.21.15.98:0/1082043966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41017f0 msgr2=0x7fbeb4075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41017f0 0x7fbeb4075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 -- 172.21.15.98:0/1082043966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb40749a0 con 0x7fbeb41033a0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb9600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbeac036040 tx=0x7fbeac0337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac043020 con 0x7fbeb41033a0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeac02ca60 con 0x7fbeb41033a0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.554+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb41a1ba0 con 0x7fbeb41033a0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.555+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac034690 con 0x7fbeb41033a0 2024-03-16T12:17:58.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.555+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb41a20f0 con 0x7fbeb41033a0 2024-03-16T12:17:58.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.556+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb4108ab0 con 0x7fbeb41033a0 2024-03-16T12:17:58.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.558+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbeac046020 con 0x7fbeb41033a0 2024-03-16T12:17:58.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.559+0000 7fbeb2000640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 0x7fbe940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:17:58.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.559+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbeac0f1430 con 0x7fbeb41033a0 2024-03-16T12:17:58.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.559+0000 7fbeb8c00640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 0x7fbe940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:17:58.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.560+0000 7fbeb8c00640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 0x7fbe940a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4102840 tx=0x7fbea4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:17:58.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.563+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbeac04d020 con 0x7fbeb41033a0 2024-03-16T12:17:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.801+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbeb402d370 con 0x7fbeb41033a0 2024-03-16T12:17:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.802+0000 7fbeb2000640 1 -- 172.21.15.98:0/1082043966 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fbeac0be8d0 con 0x7fbeb41033a0 2024-03-16T12:17:58.785 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:17:58.785 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T12:17:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-03-16T12:17:58.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 msgr2=0x7fbe940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 0x7fbe940a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4102840 tx=0x7fbea4009260 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 msgr2=0x7fbeb4074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbeac036040 tx=0x7fbeac0337b0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 shutdown_connections 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe940a2800 0x7fbe940a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb41033a0 0x7fbeb4074110 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb41029e0 0x7fbeb4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 --2- 172.21.15.98:0/1082043966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb41017f0 0x7fbeb4075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 >> 172.21.15.98:0/1082043966 conn(0x7fbeb40fcd80 msgr2=0x7fbeb4105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 shutdown_connections 2024-03-16T12:17:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:17:58.805+0000 7fbebb600640 1 -- 172.21.15.98:0/1082043966 wait complete. 2024-03-16T12:17:59.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:59.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:59.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:59.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:59.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: Deploying daemon mds.k on smithi153 2024-03-16T12:17:59.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:58 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1082043966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:59.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:59.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:59.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:59.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:59.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: Deploying daemon mds.k on smithi153 2024-03-16T12:17:59.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:58 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1082043966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:59.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:17:59.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:17:59.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:17:59.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:17:59.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: Deploying daemon mds.k on smithi153 2024-03-16T12:17:59.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:58 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1082043966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:17:59.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:17:59 smithi153 ceph-mon[36901]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:00.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:17:59 smithi098 ceph-mon[31633]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:00.255 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 systemd[1]: Starting Ceph mds.k for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:00.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:17:59 smithi173 ceph-mon[36632]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:00.633 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 podman[66697]: 2024-03-16 12:18:00.296753819 +0000 UTC m=+0.018053744 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:00.633 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 podman[66697]: 2024-03-16 12:18:00.406312374 +0000 UTC m=+0.127612284 container create 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-16T12:18:00.633 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 podman[66697]: 2024-03-16 12:18:00.625680329 +0000 UTC m=+0.346980256 container init 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:18:00.633 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 podman[66697]: 2024-03-16 12:18:00.628196448 +0000 UTC m=+0.349496351 container start 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T12:18:00.634 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 ceph-mds[66716]: -- 172.21.15.153:0/921239807 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fda76eba40 con 0x55fda76be800 2024-03-16T12:18:00.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:01.005 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k[66712]: starting mds.k at 2024-03-16T12:18:01.005 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 bash[66697]: 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 2024-03-16T12:18:01.005 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:18:00 smithi153 systemd[1]: Started Ceph mds.k for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 -- 172.21.15.98:0/4063686665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95441055b0 msgr2=0x7f954410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 --2- 172.21.15.98:0/4063686665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95441055b0 0x7f954410be40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f953c009ca0 tx=0x7f953c02c100 comp rx=0 tx=0).stop 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 -- 172.21.15.98:0/4063686665 shutdown_connections 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 --2- 172.21.15.98:0/4063686665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95441055b0 0x7f954410be40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 --2- 172.21.15.98:0/4063686665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9544104bf0 0x7f9544105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 --2- 172.21.15.98:0/4063686665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.033+0000 7f9549400640 1 -- 172.21.15.98:0/4063686665 >> 172.21.15.98:0/4063686665 conn(0x7f95440fef90 msgr2=0x7f95441013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.034+0000 7f9549400640 1 -- 172.21.15.98:0/4063686665 shutdown_connections 2024-03-16T12:18:01.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.034+0000 7f9549400640 1 -- 172.21.15.98:0/4063686665 wait complete. 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 Processor -- start 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 -- start start 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 0x7f954419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95441055b0 0x7f95441a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f954410f680 con 0x7f9544103a00 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f954410f500 con 0x7f9544104bf0 2024-03-16T12:18:01.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9549400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f954410f800 con 0x7f95441055b0 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f953be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 0x7f954419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f9542800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f953be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 0x7f954419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:52280/0 (socket says 172.21.15.98:52280) 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.035+0000 7f953be00640 1 -- 172.21.15.98:0/1015556233 learned_addr learned my addr 172.21.15.98:0/1015556233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9543200640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95441055b0 0x7f95441a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 -- 172.21.15.98:0/1015556233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95441055b0 msgr2=0x7f95441a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95441055b0 0x7f95441a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 -- 172.21.15.98:0/1015556233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 msgr2=0x7f954419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 0x7f954419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 -- 172.21.15.98:0/1015556233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95441a18d0 con 0x7f9544103a00 2024-03-16T12:18:01.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9542800640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544199c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f953400e9c0 tx=0x7f953400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400ccc0 con 0x7f9544103a00 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95340035f0 con 0x7f9544103a00 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95441a1bc0 con 0x7f9544103a00 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.036+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400ce60 con 0x7f9544103a00 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.037+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95441a2110 con 0x7f9544103a00 2024-03-16T12:18:01.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.038+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f954410ad20 con 0x7f9544103a00 2024-03-16T12:18:01.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.040+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f95340037a0 con 0x7f9544103a00 2024-03-16T12:18:01.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.041+0000 7f953b400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 0x7f95280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:01.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.041+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f95340cea00 con 0x7f9544103a00 2024-03-16T12:18:01.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.041+0000 7f953be00640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 0x7f95280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:01.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.042+0000 7f953be00640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 0x7f95280a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f954419b190 tx=0x7f952c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:01.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.045+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f953409bea0 con 0x7f9544103a00 2024-03-16T12:18:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.284+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f954402d1b0 con 0x7f9544103a00 2024-03-16T12:18:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.285+0000 7f953b400640 1 -- 172.21.15.98:0/1015556233 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f953409b7a0 con 0x7f9544103a00 2024-03-16T12:18:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T12:18:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 msgr2=0x7f95280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 0x7f95280a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f954419b190 tx=0x7f952c005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 msgr2=0x7f9544199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544199c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f953400e9c0 tx=0x7f953400ee80 comp rx=0 tx=0).stop 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 shutdown_connections 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f95280a2800 0x7f95280a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95441055b0 0x7f95441a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9544104bf0 0x7f954419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 --2- 172.21.15.98:0/1015556233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9544103a00 0x7f9544199c90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 >> 172.21.15.98:0/1015556233 conn(0x7f95440fef90 msgr2=0x7f9544107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 shutdown_connections 2024-03-16T12:18:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:01.288+0000 7f9549400640 1 -- 172.21.15.98:0/1015556233 wait complete. 2024-03-16T12:18:02.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:02.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: Deploying daemon mds.j on smithi098 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6832/3502700465,v1:172.21.15.153:6833/3502700465] up:boot 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: fsmap 2 up:standby 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T12:18:02.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:01 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1015556233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:02.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: Deploying daemon mds.j on smithi098 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6832/3502700465,v1:172.21.15.153:6833/3502700465] up:boot 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: fsmap 2 up:standby 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T12:18:02.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:01 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1015556233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:02.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:02.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: Deploying daemon mds.j on smithi098 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6832/3502700465,v1:172.21.15.153:6833/3502700465] up:boot 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: fsmap 2 up:standby 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T12:18:02.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:01 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1015556233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:03.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 -- 172.21.15.98:0/1634686190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c110060 msgr2=0x7f593c112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 --2- 172.21.15.98:0/1634686190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c110060 0x7f593c112450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5934009be0 tx=0x7f593402bef0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 -- 172.21.15.98:0/1634686190 shutdown_connections 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 --2- 172.21.15.98:0/1634686190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c110060 0x7f593c112450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 --2- 172.21.15.98:0/1634686190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c109140 0x7f593c10fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 --2- 172.21.15.98:0/1634686190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c1087e0 0x7f593c108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 -- 172.21.15.98:0/1634686190 >> 172.21.15.98:0/1634686190 conn(0x7f593c06e640 msgr2=0x7f593c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 -- 172.21.15.98:0/1634686190 shutdown_connections 2024-03-16T12:18:03.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.533+0000 7f5942400640 1 -- 172.21.15.98:0/1634686190 wait complete. 2024-03-16T12:18:03.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.534+0000 7f5942400640 1 Processor -- start 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.534+0000 7f5942400640 1 -- start start 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c1087e0 0x7f593c1a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 0x7f593c1a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 0x7f593c1a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f593c117ed0 con 0x7f593c109140 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f593c117d50 con 0x7f593c110060 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f5942400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f593c118050 con 0x7f593c1087e0 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 0x7f593c1a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 0x7f593c1a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 0x7f593c1a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57470/0 (socket says 172.21.15.98:57470) 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593aa00640 1 -- 172.21.15.98:0/2085341402 learned_addr learned my addr 172.21.15.98:0/2085341402 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593b400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c1087e0 0x7f593c1a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.535+0000 7f593be00640 1 -- 172.21.15.98:0/2085341402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c1087e0 msgr2=0x7f593c1a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f593be00640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c1087e0 0x7f593c1a2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f593be00640 1 -- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 msgr2=0x7f593c1a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f593be00640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 0x7f593c1a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f593be00640 1 -- 172.21.15.98:0/2085341402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c1aa0a0 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f593be00640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 0x7f593c1a9a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5934036040 tx=0x7f59340337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934043020 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f593402ca60 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c1aa330 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934034690 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.536+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c1aa850 con 0x7f593c110060 2024-03-16T12:18:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.537+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f593c1aaa50 con 0x7f593c110060 2024-03-16T12:18:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.539+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5934046020 con 0x7f593c110060 2024-03-16T12:18:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.540+0000 7f5933e00640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 0x7f59200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.541+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59340f0c00 con 0x7f593c110060 2024-03-16T12:18:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.541+0000 7f593b400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 0x7f59200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:03.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.541+0000 7f593b400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 0x7f59200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f592c009bd0 tx=0x7f592c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:03.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.545+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f593404d020 con 0x7f593c110060 2024-03-16T12:18:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.786+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f593c1aade0 con 0x7f593c110060 2024-03-16T12:18:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.786+0000 7f5933e00640 1 -- 172.21.15.98:0/2085341402 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f59340be0a0 con 0x7f593c110060 2024-03-16T12:18:03.769 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:03.769 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T12:18:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 msgr2=0x7f59200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 0x7f59200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f592c009bd0 tx=0x7f592c005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 msgr2=0x7f593c1a9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 0x7f593c1a9a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5934036040 tx=0x7f59340337b0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 shutdown_connections 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f59200a2800 0x7f59200a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c110060 0x7f593c1a9a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f593c109140 0x7f593c1a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 --2- 172.21.15.98:0/2085341402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f593c1087e0 0x7f593c1a2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 >> 172.21.15.98:0/2085341402 conn(0x7f593c06e640 msgr2=0x7f593c070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 shutdown_connections 2024-03-16T12:18:03.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:03.789+0000 7f5942400640 1 -- 172.21.15.98:0/2085341402 wait complete. 2024-03-16T12:18:03.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:03 smithi098 ceph-mon[31633]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:03.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:03 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2085341402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:04.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:03 smithi173 ceph-mon[36632]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:04.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:03 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2085341402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:04.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:03 smithi153 ceph-mon[36901]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:04.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:03 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2085341402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:04.347 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 podman[72276]: 2024-03-16 12:18:04.107848691 +0000 UTC m=+0.157335944 container create b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T12:18:04.347 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 podman[72276]: 2024-03-16 12:18:04.362837584 +0000 UTC m=+0.412324874 container init b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T12:18:04.744 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 podman[72276]: 2024-03-16 12:18:04.372952209 +0000 UTC m=+0.422439444 container start b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:18:04.744 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 ceph-mds[72294]: -- 172.21.15.98:0/2331054839 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b175d5a40 con 0x563b175a8800 2024-03-16T12:18:04.744 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j[72290]: starting mds.j at 2024-03-16T12:18:04.744 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 bash[72276]: b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 2024-03-16T12:18:04.744 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:18:04 smithi098 systemd[1]: Started Ceph mds.j for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:05.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: Deploying daemon mds.d on smithi098 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:05.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.98:6834/211337722,v1:172.21.15.98:6835/211337722] up:boot 2024-03-16T12:18:05.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: fsmap 3 up:standby 2024-03-16T12:18:05.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:05 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T12:18:06.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: Deploying daemon mds.d on smithi098 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.98:6834/211337722,v1:172.21.15.98:6835/211337722] up:boot 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: fsmap 3 up:standby 2024-03-16T12:18:06.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:05 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 -- 172.21.15.98:0/391594352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc100820 msgr2=0x7f27fc100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 --2- 172.21.15.98:0/391594352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc100820 0x7f27fc100c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009860 tx=0x7f27e802bb70 comp rx=0 tx=0).stop 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 -- 172.21.15.98:0/391594352 shutdown_connections 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 --2- 172.21.15.98:0/391594352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc1011c0 0x7f27fc101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 --2- 172.21.15.98:0/391594352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc100820 0x7f27fc100c80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 --2- 172.21.15.98:0/391594352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc0ffec0 0x7f27fc1002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 -- 172.21.15.98:0/391594352 >> 172.21.15.98:0/391594352 conn(0x7f27fc0fba60 msgr2=0x7f27fc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 -- 172.21.15.98:0/391594352 shutdown_connections 2024-03-16T12:18:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.028+0000 7f2801400640 1 -- 172.21.15.98:0/391594352 wait complete. 2024-03-16T12:18:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.029+0000 7f2801400640 1 Processor -- start 2024-03-16T12:18:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.029+0000 7f2801400640 1 -- start start 2024-03-16T12:18:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.029+0000 7f2801400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 0x7f27fc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.029+0000 7f2801400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57506/0 (socket says 172.21.15.98:57506) 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 0x7f27fc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 0x7f27fc199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:60900/0 (socket says 172.21.15.98:60900) 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2801400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 0x7f27fc1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2801400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc10f450 con 0x7f27fc100820 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc10f2d0 con 0x7f27fc1011c0 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 -- 172.21.15.98:0/1080070150 learned_addr learned my addr 172.21.15.98:0/1080070150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc10f5d0 con 0x7f27fc0ffec0 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2800a00640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 0x7f27fc1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 -- 172.21.15.98:0/1080070150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 msgr2=0x7f27fc199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 0x7f27fc199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 -- 172.21.15.98:0/1080070150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 msgr2=0x7f27fc1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:06.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 0x7f27fc1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f27fb400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc1a16f0 con 0x7f27fc100820 2024-03-16T12:18:06.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.030+0000 7f2800a00640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 0x7f27fc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:18:06.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f27fb400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27e80349e0 tx=0x7f27e8032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:06.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8041030 con 0x7f27fc100820 2024-03-16T12:18:06.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27fc1a1980 con 0x7f27fc100820 2024-03-16T12:18:06.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e802c510 con 0x7f27fc100820 2024-03-16T12:18:06.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8039040 con 0x7f27fc100820 2024-03-16T12:18:06.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.031+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27fc1a1e70 con 0x7f27fc100820 2024-03-16T12:18:06.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.033+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27fc10ac10 con 0x7f27fc100820 2024-03-16T12:18:06.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.035+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f27e8044070 con 0x7f27fc100820 2024-03-16T12:18:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.037+0000 7f27f8c00640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 0x7f27e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.037+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27e80efc70 con 0x7f27fc100820 2024-03-16T12:18:06.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.037+0000 7f27fbe00640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 0x7f27e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.038+0000 7f27fbe00640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 0x7f27e00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009bd0 tx=0x7f27ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.042+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27e80bd110 con 0x7f27fc100820 2024-03-16T12:18:06.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: Deploying daemon mds.d on smithi098 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.98:6834/211337722,v1:172.21.15.98:6835/211337722] up:boot 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: fsmap 3 up:standby 2024-03-16T12:18:06.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:05 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T12:18:06.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.283+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27fc02d1b0 con 0x7f27fc100820 2024-03-16T12:18:06.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.284+0000 7f27f8c00640 1 -- 172.21.15.98:0/1080070150 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f27e80bca10 con 0x7f27fc100820 2024-03-16T12:18:06.266 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:06.266 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T12:18:06.267 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 4 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.286+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 msgr2=0x7f27e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 0x7f27e00a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009bd0 tx=0x7f27ec005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 msgr2=0x7f27fc19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27e80349e0 tx=0x7f27e8032450 comp rx=0 tx=0).stop 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 shutdown_connections 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f27e00a2800 0x7f27e00a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27fc1011c0 0x7f27fc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100820 0x7f27fc19a010 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 --2- 172.21.15.98:0/1080070150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27fc0ffec0 0x7f27fc199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:06.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 >> 172.21.15.98:0/1080070150 conn(0x7f27fc0fba60 msgr2=0x7f27fc0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:06.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 shutdown_connections 2024-03-16T12:18:06.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:06.287+0000 7f2801400640 1 -- 172.21.15.98:0/1080070150 wait complete. 2024-03-16T12:18:06.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:06 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1080070150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:07.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:06 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1080070150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:07.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:06 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1080070150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:07.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:07 smithi098 ceph-mon[31633]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:07.994 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:07 smithi098 systemd[1]: Starting Ceph mds.d for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:08.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:07 smithi153 ceph-mon[36901]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:08.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:07 smithi173 ceph-mon[36632]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:08.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:08.359 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 podman[72623]: 2024-03-16 12:18:08.016451947 +0000 UTC m=+0.033442898 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:08.359 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 podman[72623]: 2024-03-16 12:18:08.148843369 +0000 UTC m=+0.165834357 container create a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-16T12:18:08.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 -- 172.21.15.98:0/2832627365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41027e0 msgr2=0x7ff4c4102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:08.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 --2- 172.21.15.98:0/2832627365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41027e0 0x7ff4c4102c60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009900 tx=0x7ff4b002bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 -- 172.21.15.98:0/2832627365 shutdown_connections 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 --2- 172.21.15.98:0/2832627365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41031a0 0x7ff4c4109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 --2- 172.21.15.98:0/2832627365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41027e0 0x7ff4c4102c60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 --2- 172.21.15.98:0/2832627365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41015f0 0x7ff4c4101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 -- 172.21.15.98:0/2832627365 >> 172.21.15.98:0/2832627365 conn(0x7ff4c40fcb40 msgr2=0x7ff4c40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 -- 172.21.15.98:0/2832627365 shutdown_connections 2024-03-16T12:18:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.506+0000 7ff4c9400640 1 -- 172.21.15.98:0/2832627365 wait complete. 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.507+0000 7ff4c9400640 1 Processor -- start 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.507+0000 7ff4c9400640 1 -- start start 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41015f0 0x7ff4c4199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41027e0 0x7ff4c419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c410d2b0 con 0x7ff4c41031a0 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c410d130 con 0x7ff4c41027e0 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c9400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c410d430 con 0x7ff4c41015f0 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:08.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49772/0 (socket says 172.21.15.98:49772) 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 -- 172.21.15.98:0/500878593 learned_addr learned my addr 172.21.15.98:0/500878593 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c3e00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41015f0 0x7ff4c4199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c3400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41027e0 0x7ff4c419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 -- 172.21.15.98:0/500878593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41015f0 msgr2=0x7ff4c4199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41015f0 0x7ff4c4199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 -- 172.21.15.98:0/500878593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41027e0 msgr2=0x7ff4c419a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41027e0 0x7ff4c419a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.508+0000 7ff4c8a00640 1 -- 172.21.15.98:0/500878593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c41a1780 con 0x7ff4c41031a0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c8a00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00eaa0 tx=0x7ff4bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00cdb0 con 0x7ff4c41031a0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4bc005020 con 0x7ff4c41031a0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c41a1a70 con 0x7ff4c41031a0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0116b0 con 0x7ff4c41031a0 2024-03-16T12:18:08.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.509+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c41a1f40 con 0x7ff4c41031a0 2024-03-16T12:18:08.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.510+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c41a2230 con 0x7ff4c41031a0 2024-03-16T12:18:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.514+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff4bc005560 con 0x7ff4c41031a0 2024-03-16T12:18:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.515+0000 7ff4c0c00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 0x7ff4a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.515+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff4bc0cebc0 con 0x7ff4c41031a0 2024-03-16T12:18:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.516+0000 7ff4c3e00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 0x7ff4a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.516+0000 7ff4c3e00640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 0x7ff4a80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40095f0 tx=0x7ff4b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.519+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff4bc016030 con 0x7ff4c41031a0 2024-03-16T12:18:08.744 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 podman[72623]: 2024-03-16 12:18:08.419416896 +0000 UTC m=+0.436407869 container init a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-16T12:18:08.745 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 podman[72623]: 2024-03-16 12:18:08.427978282 +0000 UTC m=+0.444969268 container start a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:18:08.745 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 ceph-mds[72663]: -- 172.21.15.98:0/4282288258 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5603c0779a40 con 0x5603c074c800 2024-03-16T12:18:08.745 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d[72639]: starting mds.d at 2024-03-16T12:18:08.745 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 bash[72623]: a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 2024-03-16T12:18:08.745 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:18:08 smithi098 systemd[1]: Started Ceph mds.d for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:08.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.778+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4c41a2400 con 0x7ff4c41031a0 2024-03-16T12:18:08.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.779+0000 7ff4c0c00640 1 -- 172.21.15.98:0/500878593 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7ff4bc09c0e0 con 0x7ff4c41031a0 2024-03-16T12:18:08.761 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:08.761 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-16T12:18:08.761 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-03-16T12:18:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 msgr2=0x7ff4a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 0x7ff4a80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40095f0 tx=0x7ff4b4005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 msgr2=0x7ff4c41a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00eaa0 tx=0x7ff4bc00ef60 comp rx=0 tx=0).stop 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 shutdown_connections 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff4a80a2800 0x7ff4a80a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4c41031a0 0x7ff4c41a1110 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4c41027e0 0x7ff4c419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 --2- 172.21.15.98:0/500878593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4c41015f0 0x7ff4c4199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.782+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 >> 172.21.15.98:0/500878593 conn(0x7ff4c40fcb40 msgr2=0x7ff4c4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.783+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 shutdown_connections 2024-03-16T12:18:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:08.783+0000 7ff4c9400640 1 -- 172.21.15.98:0/500878593 wait complete. 2024-03-16T12:18:09.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.98:6836/3487564371,v1:172.21.15.98:6837/3487564371] up:boot 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: fsmap 4 up:standby 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: Deploying daemon mds.e on smithi153 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:09.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:09 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/500878593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.98:6836/3487564371,v1:172.21.15.98:6837/3487564371] up:boot 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: fsmap 4 up:standby 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:10.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:10.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:10.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: Deploying daemon mds.e on smithi153 2024-03-16T12:18:10.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:10.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:09 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/500878593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:10.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.98:6836/3487564371,v1:172.21.15.98:6837/3487564371] up:boot 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: fsmap 4 up:standby 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: Deploying daemon mds.e on smithi153 2024-03-16T12:18:10.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:09 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/500878593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:10.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.020+0000 7feabca00640 1 -- 172.21.15.98:0/3371198540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 msgr2=0x7feab8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.020+0000 7feabca00640 1 --2- 172.21.15.98:0/3371198540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab8109470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009860 tx=0x7feaa002bb90 comp rx=0 tx=0).stop 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 -- 172.21.15.98:0/3371198540 shutdown_connections 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 --2- 172.21.15.98:0/3371198540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 --2- 172.21.15.98:0/3371198540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab8109470 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 --2- 172.21.15.98:0/3371198540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 -- 172.21.15.98:0/3371198540 >> 172.21.15.98:0/3371198540 conn(0x7feab8076dc0 msgr2=0x7feab80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:11.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 -- 172.21.15.98:0/3371198540 shutdown_connections 2024-03-16T12:18:11.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.021+0000 7feabca00640 1 -- 172.21.15.98:0/3371198540 wait complete. 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 Processor -- start 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 -- start start 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab81a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113a80 con 0x7feab8108ff0 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113900 con 0x7feab8107e00 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feabca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113c00 con 0x7feab81099b0 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feab5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:11.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feab5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:56736/0 (socket says 172.21.15.98:56736) 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feab5e00640 1 -- 172.21.15.98:0/3101121842 learned_addr learned my addr 172.21.15.98:0/3101121842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.022+0000 7feaad400640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab6800640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab81a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 -- 172.21.15.98:0/3101121842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 msgr2=0x7feab81a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab81a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 -- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 msgr2=0x7feab819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 -- 172.21.15.98:0/3101121842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab81a5d40 con 0x7feab8107e00 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab6800640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab81a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feaad400640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feab5e00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e9a0 tx=0x7feaa800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa800cd80 con 0x7feab8107e00 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaa8010910 con 0x7feab8107e00 2024-03-16T12:18:11.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab81a5fd0 con 0x7feab8107e00 2024-03-16T12:18:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8018a30 con 0x7feab8107e00 2024-03-16T12:18:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.023+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab81a6490 con 0x7feab8107e00 2024-03-16T12:18:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.025+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab810f0c0 con 0x7feab8107e00 2024-03-16T12:18:11.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.026+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7feaa8010ac0 con 0x7feab8107e00 2024-03-16T12:18:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.027+0000 7feaaf400640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 0x7fea9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.028+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feaa80ceb10 con 0x7feab8107e00 2024-03-16T12:18:11.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.028+0000 7feaad400640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 0x7fea9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:11.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.028+0000 7feaad400640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 0x7fea9c0a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009830 tx=0x7feaa0005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.032+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feaa8053740 con 0x7feab8107e00 2024-03-16T12:18:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.269+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feab802d1b0 con 0x7feab8107e00 2024-03-16T12:18:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.270+0000 7feaaf400640 1 -- 172.21.15.98:0/3101121842 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7feaa809bfb0 con 0x7feab8107e00 2024-03-16T12:18:11.252 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:11.252 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-16T12:18:11.253 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-03-16T12:18:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 msgr2=0x7fea9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 0x7fea9c0a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009830 tx=0x7feaa0005d20 comp rx=0 tx=0).stop 2024-03-16T12:18:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 msgr2=0x7feab819e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e9a0 tx=0x7feaa800ee60 comp rx=0 tx=0).stop 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 shutdown_connections 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fea9c0a2800 0x7fea9c0a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feab81099b0 0x7feab81a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feab8108ff0 0x7feab819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 --2- 172.21.15.98:0/3101121842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab8107e00 0x7feab819e120 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 >> 172.21.15.98:0/3101121842 conn(0x7feab8076dc0 msgr2=0x7feab810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 shutdown_connections 2024-03-16T12:18:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:11.273+0000 7feabca00640 1 -- 172.21.15.98:0/3101121842 wait complete. 2024-03-16T12:18:11.256 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 systemd[1]: Starting Ceph mds.e for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:11.562 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 podman[67017]: 2024-03-16 12:18:11.186918259 +0000 UTC m=+0.018218419 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:11.563 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 podman[67017]: 2024-03-16 12:18:11.339878017 +0000 UTC m=+0.171178162 container create 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:18:11.563 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 podman[67017]: 2024-03-16 12:18:11.580877547 +0000 UTC m=+0.412177697 container init 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-16T12:18:12.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:12.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3101121842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:12.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 podman[67017]: 2024-03-16 12:18:11.58351157 +0000 UTC m=+0.414811717 container start 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-mds[67036]: -- 172.21.15.153:0/1965376227 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd07047a40 con 0x55bd0701ac00 2024-03-16T12:18:12.006 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: starting mds.e at 2024-03-16T12:18:12.007 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 bash[67017]: 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 2024-03-16T12:18:12.007 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:18:11 smithi153 systemd[1]: Started Ceph mds.e for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:12.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:12.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3101121842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:12.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:11 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:12.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3101121842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:12.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:11 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:13.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:12 smithi173 ceph-mon[36632]: Deploying daemon mds.f on smithi173 2024-03-16T12:18:13.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:12 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6834/4013234440,v1:172.21.15.153:6835/4013234440] up:boot 2024-03-16T12:18:13.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:12 smithi173 ceph-mon[36632]: fsmap 5 up:standby 2024-03-16T12:18:13.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:12 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:18:13.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:12 smithi098 ceph-mon[31633]: Deploying daemon mds.f on smithi173 2024-03-16T12:18:13.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:12 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6834/4013234440,v1:172.21.15.153:6835/4013234440] up:boot 2024-03-16T12:18:13.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:12 smithi098 ceph-mon[31633]: fsmap 5 up:standby 2024-03-16T12:18:13.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:12 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:18:13.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:12 smithi153 ceph-mon[36901]: Deploying daemon mds.f on smithi173 2024-03-16T12:18:13.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:12 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6834/4013234440,v1:172.21.15.153:6835/4013234440] up:boot 2024-03-16T12:18:13.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:12 smithi153 ceph-mon[36901]: fsmap 5 up:standby 2024-03-16T12:18:13.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:12 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:18:13.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 -- 172.21.15.98:0/448680591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb948100700 msgr2=0x7fb948100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 --2- 172.21.15.98:0/448680591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb948100700 0x7fb948100b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb938009ca0 tx=0x7fb93802c100 comp rx=0 tx=0).stop 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 -- 172.21.15.98:0/448680591 shutdown_connections 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 --2- 172.21.15.98:0/448680591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481022b0 0x7fb948108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 --2- 172.21.15.98:0/448680591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9481018f0 0x7fb948101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 --2- 172.21.15.98:0/448680591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb948100700 0x7fb948100b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 -- 172.21.15.98:0/448680591 >> 172.21.15.98:0/448680591 conn(0x7fb9480fbc70 msgr2=0x7fb9480fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 -- 172.21.15.98:0/448680591 shutdown_connections 2024-03-16T12:18:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.514+0000 7fb94e600640 1 -- 172.21.15.98:0/448680591 wait complete. 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.515+0000 7fb94e600640 1 Processor -- start 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 -- start start 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 0x7fb948198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 0x7fb94819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb94810d460 con 0x7fb9481022b0 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb94810d2e0 con 0x7fb9481018f0 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb94e600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb94810d5e0 con 0x7fb948100700 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb947e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 0x7fb94819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb947400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb947400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:37216/0 (socket says 172.21.15.98:37216) 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb947400640 1 -- 172.21.15.98:0/2193773338 learned_addr learned my addr 172.21.15.98:0/2193773338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.516+0000 7fb946a00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 0x7fb948198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 -- 172.21.15.98:0/2193773338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 msgr2=0x7fb948198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 0x7fb948198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 -- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 msgr2=0x7fb94819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 0x7fb94819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 -- 172.21.15.98:0/2193773338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94819f720 con 0x7fb948100700 2024-03-16T12:18:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb946a00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 0x7fb948198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947e00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 0x7fb94819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb947400640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9380324c0 tx=0x7fb938035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938039040 con 0x7fb948100700 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94819f9b0 con 0x7fb948100700 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb938044070 con 0x7fb948100700 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9380044a0 con 0x7fb948100700 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.517+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94819fef0 con 0x7fb948100700 2024-03-16T12:18:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.518+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb948107aa0 con 0x7fb948100700 2024-03-16T12:18:13.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.520+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb938004640 con 0x7fb948100700 2024-03-16T12:18:13.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.522+0000 7fb93fe00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 0x7fb92c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.522+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb9380f0cf0 con 0x7fb948100700 2024-03-16T12:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.522+0000 7fb946a00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 0x7fb92c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.522+0000 7fb946a00640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 0x7fb92c0a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb948199010 tx=0x7fb930005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:13.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.526+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9380be190 con 0x7fb948100700 2024-03-16T12:18:13.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.760+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb94802d1b0 con 0x7fb948100700 2024-03-16T12:18:13.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.761+0000 7fb93fe00640 1 -- 172.21.15.98:0/2193773338 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7fb9380bda90 con 0x7fb948100700 2024-03-16T12:18:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:13.743 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T12:18:13.743 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.763+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 msgr2=0x7fb92c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.763+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 0x7fb92c0a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb948199010 tx=0x7fb930005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.763+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 msgr2=0x7fb948197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.763+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9380324c0 tx=0x7fb938035d90 comp rx=0 tx=0).stop 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 shutdown_connections 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb92c0a2800 0x7fb92c0a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481022b0 0x7fb94819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9481018f0 0x7fb948198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 --2- 172.21.15.98:0/2193773338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb948100700 0x7fb948197b00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 >> 172.21.15.98:0/2193773338 conn(0x7fb9480fbc70 msgr2=0x7fb948104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 shutdown_connections 2024-03-16T12:18:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:13.764+0000 7fb94e600640 1 -- 172.21.15.98:0/2193773338 wait complete. 2024-03-16T12:18:14.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:13 smithi173 ceph-mon[36632]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:14.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:13 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2193773338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:14.072 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 systemd[1]: Starting Ceph mds.f for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:14.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:13 smithi098 ceph-mon[31633]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:14.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:13 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2193773338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:14.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:13 smithi153 ceph-mon[36901]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:14.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:13 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2193773338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:14.520 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 podman[66883]: 2024-03-16 12:18:14.156548187 +0000 UTC m=+0.017762589 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:14.520 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 podman[66883]: 2024-03-16 12:18:14.268791388 +0000 UTC m=+0.130005828 container create 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-03-16T12:18:14.521 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 podman[66883]: 2024-03-16 12:18:14.538262394 +0000 UTC m=+0.399476797 container init 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-16T12:18:14.821 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 podman[66883]: 2024-03-16 12:18:14.541052106 +0000 UTC m=+0.402266497 container start 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T12:18:14.821 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 ceph-mds[66902]: -- 172.21.15.173:0/2928650975 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8d785da40 con 0x55d8d7830c00 2024-03-16T12:18:14.821 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: starting mds.f at 2024-03-16T12:18:14.821 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 bash[66883]: 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b 2024-03-16T12:18:14.822 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:18:14 smithi173 systemd[1]: Started Ceph mds.f for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:15.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.005+0000 7fd19d200640 1 -- 172.21.15.98:0/3007189461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 msgr2=0x7fd198107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.005+0000 7fd19d200640 1 --2- 172.21.15.98:0/3007189461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd198107080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd180009900 tx=0x7fd18002bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 -- 172.21.15.98:0/3007189461 shutdown_connections 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 --2- 172.21.15.98:0/3007189461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1981075c0 0x7fd1981099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 --2- 172.21.15.98:0/3007189461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd198107080 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 --2- 172.21.15.98:0/3007189461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd198102360 0x7fd198104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 -- 172.21.15.98:0/3007189461 >> 172.21.15.98:0/3007189461 conn(0x7fd1980fbc90 msgr2=0x7fd1980fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 -- 172.21.15.98:0/3007189461 shutdown_connections 2024-03-16T12:18:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.006+0000 7fd19d200640 1 -- 172.21.15.98:0/3007189461 wait complete. 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 Processor -- start 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 -- start start 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd198102360 0x7fd198197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1981075c0 0x7fd19819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd19810d4d0 con 0x7fd198104c90 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd19810d350 con 0x7fd198102360 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd19d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd19810d650 con 0x7fd1981075c0 2024-03-16T12:18:15.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd195400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd195400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49832/0 (socket says 172.21.15.98:49832) 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd195400640 1 -- 172.21.15.98:0/673445167 learned_addr learned my addr 172.21.15.98:0/673445167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.007+0000 7fd195e00640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd198102360 0x7fd198197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd196800640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1981075c0 0x7fd19819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 -- 172.21.15.98:0/673445167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1981075c0 msgr2=0x7fd19819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1981075c0 0x7fd19819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 -- 172.21.15.98:0/673445167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd198102360 msgr2=0x7fd198197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd198102360 0x7fd198197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 -- 172.21.15.98:0/673445167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19819f740 con 0x7fd198104c90 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd195400640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd180000c00 tx=0x7fd18002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180043020 con 0x7fd198104c90 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1800054e0 con 0x7fd198104c90 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19819f9d0 con 0x7fd198104c90 2024-03-16T12:18:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.008+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180003c40 con 0x7fd198104c90 2024-03-16T12:18:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.009+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19819ff70 con 0x7fd198104c90 2024-03-16T12:18:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.010+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1980fd780 con 0x7fd198104c90 2024-03-16T12:18:15.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: Deploying daemon mds.c on smithi173 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6834/2799668647,v1:172.21.15.173:6835/2799668647] up:boot 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: fsmap 6 up:standby 2024-03-16T12:18:15.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:15 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:18:15.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.012+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd180046020 con 0x7fd198104c90 2024-03-16T12:18:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.013+0000 7fd18ea00640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 0x7fd17c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.013+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd1800f05f0 con 0x7fd198104c90 2024-03-16T12:18:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.014+0000 7fd195e00640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 0x7fd17c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.014+0000 7fd195e00640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 0x7fd17c0a4dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd188005fd0 tx=0x7fd188009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:16.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.018+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd18004d020 con 0x7fd198104c90 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: Deploying daemon mds.c on smithi173 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:16.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6834/2799668647,v1:172.21.15.173:6835/2799668647] up:boot 2024-03-16T12:18:16.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: fsmap 6 up:standby 2024-03-16T12:18:16.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:15 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:18:16.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:16.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:16.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:16.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:16.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: Deploying daemon mds.c on smithi173 2024-03-16T12:18:16.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:16.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6834/2799668647,v1:172.21.15.173:6835/2799668647] up:boot 2024-03-16T12:18:16.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: fsmap 6 up:standby 2024-03-16T12:18:16.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:15 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:18:16.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.252+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1980623e0 con 0x7fd198104c90 2024-03-16T12:18:16.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.254+0000 7fd18ea00640 1 -- 172.21.15.98:0/673445167 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fd1800bda90 con 0x7fd198104c90 2024-03-16T12:18:16.238 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:16.238 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-16T12:18:16.238 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 msgr2=0x7fd17c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 0x7fd17c0a4dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd188005fd0 tx=0x7fd188009500 comp rx=0 tx=0).stop 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 msgr2=0x7fd1981980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd180000c00 tx=0x7fd18002c650 comp rx=0 tx=0).stop 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 shutdown_connections 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd17c0a2900 0x7fd17c0a4dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1981075c0 0x7fd19819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd198104c90 0x7fd1981980b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 --2- 172.21.15.98:0/673445167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd198102360 0x7fd198197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 >> 172.21.15.98:0/673445167 conn(0x7fd1980fbc90 msgr2=0x7fd198102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 shutdown_connections 2024-03-16T12:18:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:16.257+0000 7fd19d200640 1 -- 172.21.15.98:0/673445167 wait complete. 2024-03-16T12:18:16.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:16 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/673445167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:17.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:16 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/673445167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:17.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:16 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/673445167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:17.072 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:16 smithi173 systemd[1]: Starting Ceph mds.c for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:17.400 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 podman[67189]: 2024-03-16 12:18:17.052233165 +0000 UTC m=+0.018362057 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:17.400 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 podman[67189]: 2024-03-16 12:18:17.17409887 +0000 UTC m=+0.140227742 container create edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:18:17.401 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 podman[67189]: 2024-03-16 12:18:17.418535585 +0000 UTC m=+0.384664455 container init edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T12:18:17.732 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 podman[67189]: 2024-03-16 12:18:17.421545453 +0000 UTC m=+0.387674318 container start edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:18:17.732 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mds[67208]: -- 172.21.15.173:0/3917336178 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557b0fd07a40 con 0x557b0fcda800 2024-03-16T12:18:17.732 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: starting mds.c at 2024-03-16T12:18:17.732 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 bash[67189]: edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 2024-03-16T12:18:17.732 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:18:17 smithi173 systemd[1]: Started Ceph mds.c for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:17.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:17 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:18.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:17 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:18.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:17 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:18.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.498+0000 7fe62a400640 1 -- 172.21.15.98:0/3367669626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6240773e0 msgr2=0x7fe624077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.498+0000 7fe62a400640 1 --2- 172.21.15.98:0/3367669626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6240773e0 0x7fe624077840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009900 tx=0x7fe60c02bc50 comp rx=0 tx=0).stop 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 -- 172.21.15.98:0/3367669626 shutdown_connections 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 --2- 172.21.15.98:0/3367669626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe624077d80 0x7fe624078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 --2- 172.21.15.98:0/3367669626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6240773e0 0x7fe624077840 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 --2- 172.21.15.98:0/3367669626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240761a0 0x7fe6240765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 -- 172.21.15.98:0/3367669626 >> 172.21.15.98:0/3367669626 conn(0x7fe6240fef90 msgr2=0x7fe6241013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 -- 172.21.15.98:0/3367669626 shutdown_connections 2024-03-16T12:18:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.499+0000 7fe62a400640 1 -- 172.21.15.98:0/3367669626 wait complete. 2024-03-16T12:18:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 Processor -- start 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 -- start start 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6240761a0 0x7fe62419e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe624077d80 0x7fe6241a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe624113ab0 con 0x7fe624077d80 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe624113930 con 0x7fe6240761a0 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe62a400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe624113c30 con 0x7fe6240773e0 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe623e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe624077d80 0x7fe6241a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe61be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe623400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6240761a0 0x7fe62419e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe61be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49558/0 (socket says 172.21.15.98:49558) 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.500+0000 7fe61be00640 1 -- 172.21.15.98:0/3148376850 learned_addr learned my addr 172.21.15.98:0/3148376850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 -- 172.21.15.98:0/3148376850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6240761a0 msgr2=0x7fe62419e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6240761a0 0x7fe62419e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 -- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe624077d80 msgr2=0x7fe6241a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe624077d80 0x7fe6241a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 -- 172.21.15.98:0/3148376850 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6241a5e00 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe61be00640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009230 tx=0x7fe60c02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c042020 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe60c004450 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.501+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6241a6090 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.502+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c003db0 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.502+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6241a65e0 con 0x7fe6240773e0 2024-03-16T12:18:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.503+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624110510 con 0x7fe6240773e0 2024-03-16T12:18:18.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.505+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe60c045020 con 0x7fe6240773e0 2024-03-16T12:18:18.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.506+0000 7fe620c00640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 0x7fe6080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.506+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe60c0f0660 con 0x7fe6240773e0 2024-03-16T12:18:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.506+0000 7fe623400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 0x7fe6080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.507+0000 7fe623400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 0x7fe6080a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe614009bd0 tx=0x7fe614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:18.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.510+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe60c04c020 con 0x7fe6240773e0 2024-03-16T12:18:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.747+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6240623e0 con 0x7fe6240773e0 2024-03-16T12:18:18.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.748+0000 7fe620c00640 1 -- 172.21.15.98:0/3148376850 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7fe60c0bdb00 con 0x7fe6240773e0 2024-03-16T12:18:18.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:18.731 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-16T12:18:18.731 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 msgr2=0x7fe6080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 0x7fe6080a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe614009bd0 tx=0x7fe614005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 msgr2=0x7fe62419e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009230 tx=0x7fe60c02c670 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 shutdown_connections 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6080a2800 0x7fe6080a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe624077d80 0x7fe6241a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6240773e0 0x7fe62419e720 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 --2- 172.21.15.98:0/3148376850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6240761a0 0x7fe62419e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:18.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 >> 172.21.15.98:0/3148376850 conn(0x7fe6240fef90 msgr2=0x7fe6241006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:18.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 shutdown_connections 2024-03-16T12:18:18.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:18.751+0000 7fe62a400640 1 -- 172.21.15.98:0/3148376850 wait complete. 2024-03-16T12:18:18.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:18 smithi098 ceph-mon[31633]: Deploying daemon mds.h on smithi153 2024-03-16T12:18:18.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6836/4083501894,v1:172.21.15.173:6837/4083501894] up:boot 2024-03-16T12:18:18.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:18 smithi098 ceph-mon[31633]: fsmap 7 up:standby 2024-03-16T12:18:18.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:18:19.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:18 smithi153 ceph-mon[36901]: Deploying daemon mds.h on smithi153 2024-03-16T12:18:19.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6836/4083501894,v1:172.21.15.173:6837/4083501894] up:boot 2024-03-16T12:18:19.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:18 smithi153 ceph-mon[36901]: fsmap 7 up:standby 2024-03-16T12:18:19.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:18:19.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:18 smithi173 ceph-mon[36632]: Deploying daemon mds.h on smithi153 2024-03-16T12:18:19.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6836/4083501894,v1:172.21.15.173:6837/4083501894] up:boot 2024-03-16T12:18:19.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:18 smithi173 ceph-mon[36632]: fsmap 7 up:standby 2024-03-16T12:18:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:18:19.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:19 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3148376850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:19.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:19 smithi153 ceph-mon[36901]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:19.976 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:19 smithi153 systemd[1]: Starting Ceph mds.h for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:20.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:19 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3148376850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:20.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:19 smithi173 ceph-mon[36632]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:20.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:19 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3148376850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:20.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:19 smithi098 ceph-mon[31633]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:20.255 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:19 smithi153 podman[67332]: 2024-03-16 12:18:19.896232466 +0000 UTC m=+0.017596127 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:20.255 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 podman[67332]: 2024-03-16 12:18:20.028622887 +0000 UTC m=+0.149986526 container create b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T12:18:20.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 podman[67332]: 2024-03-16 12:18:20.293945829 +0000 UTC m=+0.415309489 container init b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 podman[67332]: 2024-03-16 12:18:20.296910153 +0000 UTC m=+0.418273801 container start b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 ceph-mds[67351]: -- 172.21.15.153:0/3572632364 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1caa91a40 con 0x55e1caa65000 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: starting mds.h at 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 bash[67332]: b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 2024-03-16T12:18:20.756 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:18:20 smithi153 systemd[1]: Started Ceph mds.h for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:20.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 -- 172.21.15.98:0/2963067753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8108ff0 msgr2=0x7f36a8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:20.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 --2- 172.21.15.98:0/2963067753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8108ff0 0x7f36a8109470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3698009ca0 tx=0x7f369802c100 comp rx=0 tx=0).stop 2024-03-16T12:18:20.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 -- 172.21.15.98:0/2963067753 shutdown_connections 2024-03-16T12:18:20.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 --2- 172.21.15.98:0/2963067753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a81099b0 0x7f36a8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:20.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 --2- 172.21.15.98:0/2963067753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8108ff0 0x7f36a8109470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:20.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 --2- 172.21.15.98:0/2963067753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a8107e00 0x7f36a8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:20.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 -- 172.21.15.98:0/2963067753 >> 172.21.15.98:0/2963067753 conn(0x7f36a8076dc0 msgr2=0x7f36a80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:20.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 -- 172.21.15.98:0/2963067753 shutdown_connections 2024-03-16T12:18:20.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.985+0000 7f36b0200640 1 -- 172.21.15.98:0/2963067753 wait complete. 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.986+0000 7f36b0200640 1 Processor -- start 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.986+0000 7f36b0200640 1 -- start start 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8107e00 0x7f36a819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 0x7f36a81a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:20.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36a8113a80 con 0x7f36a8108ff0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36a8113900 con 0x7f36a81099b0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36b0200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36a8113c00 con 0x7f36a8107e00 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ad800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8107e00 0x7f36a819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57046/0 (socket says 172.21.15.98:57046) 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 -- 172.21.15.98:0/2461951621 learned_addr learned my addr 172.21.15.98:0/2461951621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ae200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 0x7f36a81a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 -- 172.21.15.98:0/2461951621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8107e00 msgr2=0x7f36a819e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8107e00 0x7f36a819e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 -- 172.21.15.98:0/2461951621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 msgr2=0x7f36a81a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 0x7f36a81a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ace00640 1 -- 172.21.15.98:0/2461951621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a81a5ce0 con 0x7f36a8108ff0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.987+0000 7f36ae200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 0x7f36a81a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36ace00640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3698009dc0 tx=0x7f3698035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698039040 con 0x7f36a8108ff0 2024-03-16T12:18:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3698044070 con 0x7f36a8108ff0 2024-03-16T12:18:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a81a5f70 con 0x7f36a8108ff0 2024-03-16T12:18:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698004600 con 0x7f36a8108ff0 2024-03-16T12:18:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.988+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a81a63d0 con 0x7f36a8108ff0 2024-03-16T12:18:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.989+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a810f0c0 con 0x7f36a8108ff0 2024-03-16T12:18:20.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.991+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3698032890 con 0x7f36a8108ff0 2024-03-16T12:18:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.992+0000 7f36a6000640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 0x7f36900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.993+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36980f0d50 con 0x7f36a8108ff0 2024-03-16T12:18:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.993+0000 7f36ad800640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 0x7f36900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.993+0000 7f36ad800640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 0x7f36900a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f36a8108e50 tx=0x7f369c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:20.997+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36980be270 con 0x7f36a8108ff0 2024-03-16T12:18:21.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.238+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36a802d3d0 con 0x7f36a8108ff0 2024-03-16T12:18:21.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.239+0000 7f36a6000640 1 -- 172.21.15.98:0/2461951621 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f36980bdb70 con 0x7f36a8108ff0 2024-03-16T12:18:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:21.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T12:18:21.223 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-03-16T12:18:21.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 msgr2=0x7f36900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:21.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 0x7f36900a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f36a8108e50 tx=0x7f369c005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:21.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 msgr2=0x7f36a819e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3698009dc0 tx=0x7f3698035d90 comp rx=0 tx=0).stop 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 shutdown_connections 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f36900a2800 0x7f36900a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36a81099b0 0x7f36a81a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a8108ff0 0x7f36a819e600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 --2- 172.21.15.98:0/2461951621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a8107e00 0x7f36a819e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 >> 172.21.15.98:0/2461951621 conn(0x7f36a8076dc0 msgr2=0x7f36a810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 shutdown_connections 2024-03-16T12:18:21.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:21.242+0000 7f36b0200640 1 -- 172.21.15.98:0/2461951621 wait complete. 2024-03-16T12:18:21.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: Deploying daemon mds.i on smithi173 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:boot 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: fsmap 8 up:standby 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:18:21.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:21 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2461951621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: Deploying daemon mds.i on smithi173 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:21.755 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:boot 2024-03-16T12:18:21.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: fsmap 8 up:standby 2024-03-16T12:18:21.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:18:21.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:21 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2461951621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:21.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: Deploying daemon mds.i on smithi173 2024-03-16T12:18:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:boot 2024-03-16T12:18:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: fsmap 8 up:standby 2024-03-16T12:18:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:18:21.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:21 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2461951621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:22.821 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:22 smithi173 systemd[1]: Starting Ceph mds.i for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:23.151 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:22 smithi173 podman[67497]: 2024-03-16 12:18:22.896551485 +0000 UTC m=+0.103701093 container create 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T12:18:23.151 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:22 smithi173 podman[67497]: 2024-03-16 12:18:22.810445904 +0000 UTC m=+0.017595485 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:23.152 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 podman[67497]: 2024-03-16 12:18:23.169744028 +0000 UTC m=+0.376893610 container init 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True) 2024-03-16T12:18:23.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 -- 172.21.15.98:0/788615613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740741e0 msgr2=0x7f1e74074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/788615613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740741e0 0x7f1e74074640 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009860 tx=0x7f1e5c02bc20 comp rx=0 tx=0).stop 2024-03-16T12:18:23.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 -- 172.21.15.98:0/788615613 shutdown_connections 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/788615613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e74109680 0x7f1e7410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/788615613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740741e0 0x7f1e74074640 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/788615613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e740757c0 0x7f1e74073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 -- 172.21.15.98:0/788615613 >> 172.21.15.98:0/788615613 conn(0x7f1e740fcd80 msgr2=0x7f1e740ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 -- 172.21.15.98:0/788615613 shutdown_connections 2024-03-16T12:18:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.488+0000 7f1e7ae00640 1 -- 172.21.15.98:0/788615613 wait complete. 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.489+0000 7f1e7ae00640 1 Processor -- start 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.489+0000 7f1e7ae00640 1 -- start start 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 0x7f1e74199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e74109680 0x7f1e741a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410f650 con 0x7f1e740757c0 2024-03-16T12:18:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410f4d0 con 0x7f1e740741e0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e7ae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410f7d0 con 0x7f1e74109680 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57064/0 (socket says 172.21.15.98:57064) 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 -- 172.21.15.98:0/128121837 learned_addr learned my addr 172.21.15.98:0/128121837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e78e00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e74109680 0x7f1e741a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73e00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 0x7f1e74199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 -- 172.21.15.98:0/128121837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e74109680 msgr2=0x7f1e741a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e74109680 0x7f1e741a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 -- 172.21.15.98:0/128121837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 msgr2=0x7f1e74199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 0x7f1e74199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73400640 1 -- 172.21.15.98:0/128121837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e741a1930 con 0x7f1e740757c0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.490+0000 7f1e73e00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 0x7f1e74199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e73400640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c034a80 tx=0x7f1e5c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c041030 con 0x7f1e740757c0 2024-03-16T12:18:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c02ce50 con 0x7f1e740757c0 2024-03-16T12:18:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e741a1bc0 con 0x7f1e740757c0 2024-03-16T12:18:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c039040 con 0x7f1e740757c0 2024-03-16T12:18:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.491+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e741a2100 con 0x7f1e740757c0 2024-03-16T12:18:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.493+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e74108440 con 0x7f1e740757c0 2024-03-16T12:18:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.494+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1e5c044070 con 0x7f1e740757c0 2024-03-16T12:18:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.495+0000 7f1e70c00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 0x7f1e580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.496+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1e5c0efaa0 con 0x7f1e740757c0 2024-03-16T12:18:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.496+0000 7f1e73e00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 0x7f1e580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.497+0000 7f1e73e00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 0x7f1e580a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1e7419b1f0 tx=0x7f1e64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:23.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.500+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e5c0bcf40 con 0x7f1e740757c0 2024-03-16T12:18:23.571 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 podman[67497]: 2024-03-16 12:18:23.172384945 +0000 UTC m=+0.379534514 container start 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T12:18:23.571 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mds[67516]: -- 172.21.15.173:0/3429460835 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d0a895a40 con 0x555d0a868800 2024-03-16T12:18:23.571 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: starting mds.i at 2024-03-16T12:18:23.572 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 bash[67497]: 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 2024-03-16T12:18:23.572 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:18:23 smithi173 systemd[1]: Started Ceph mds.i for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:23.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.743+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e7402d1b0 con 0x7f1e740757c0 2024-03-16T12:18:23.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.744+0000 7f1e70c00640 1 -- 172.21.15.98:0/128121837 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f1e5c0bc840 con 0x7f1e740757c0 2024-03-16T12:18:23.727 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:23.727 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T12:18:23.728 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 msgr2=0x7f1e580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 0x7f1e580a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1e7419b1f0 tx=0x7f1e64009260 comp rx=0 tx=0).stop 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 msgr2=0x7f1e7419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c034a80 tx=0x7f1e5c0042f0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 shutdown_connections 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e580a2880 0x7f1e580a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e74109680 0x7f1e741a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740757c0 0x7f1e7419a260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 --2- 172.21.15.98:0/128121837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e740741e0 0x7f1e74199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 >> 172.21.15.98:0/128121837 conn(0x7f1e740fcd80 msgr2=0x7f1e74105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 shutdown_connections 2024-03-16T12:18:23.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:23.747+0000 7f1e7ae00640 1 -- 172.21.15.98:0/128121837 wait complete. 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:24.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:24.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:24.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/128121837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:24.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/128121837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:24.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/128121837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:24 smithi098 ceph-mon[31633]: Deploying daemon mds.g on smithi098 2024-03-16T12:18:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:24 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:boot 2024-03-16T12:18:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:24 smithi098 ceph-mon[31633]: fsmap 9 up:standby 2024-03-16T12:18:24.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:18:25.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:24 smithi153 ceph-mon[36901]: Deploying daemon mds.g on smithi098 2024-03-16T12:18:25.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:24 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:boot 2024-03-16T12:18:25.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:24 smithi153 ceph-mon[36901]: fsmap 9 up:standby 2024-03-16T12:18:25.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:18:25.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:24 smithi173 ceph-mon[36632]: Deploying daemon mds.g on smithi098 2024-03-16T12:18:25.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:24 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:boot 2024-03-16T12:18:25.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:24 smithi173 ceph-mon[36632]: fsmap 9 up:standby 2024-03-16T12:18:25.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:18:25.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 -- 172.21.15.98:0/3181803854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8274102250 msgr2=0x7f8274104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 --2- 172.21.15.98:0/3181803854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8274102250 0x7f8274104640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8268009860 tx=0x7f826802bb70 comp rx=0 tx=0).stop 2024-03-16T12:18:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 -- 172.21.15.98:0/3181803854 shutdown_connections 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 --2- 172.21.15.98:0/3181803854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82741074b0 0x7f8274109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 --2- 172.21.15.98:0/3181803854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8274104b80 0x7f8274106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 --2- 172.21.15.98:0/3181803854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8274102250 0x7f8274104640 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8268009860 tx=0x7f826802bb70 comp rx=0 tx=0).stop 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 -- 172.21.15.98:0/3181803854 >> 172.21.15.98:0/3181803854 conn(0x7f82740fbc40 msgr2=0x7f82740fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 -- 172.21.15.98:0/3181803854 shutdown_connections 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.162+0000 7f827c600640 1 -- 172.21.15.98:0/3181803854 wait complete. 2024-03-16T12:18:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 Processor -- start 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 -- start start 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8274104b80 0x7f8274197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82741074b0 0x7f8274198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d450 con 0x7f8274104b80 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d2d0 con 0x7f82741074b0 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.164+0000 7f827c600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d5d0 con 0x7f827419d290 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.165+0000 7f827a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.165+0000 7f8279c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8274104b80 0x7f8274197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.165+0000 7f827a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49610/0 (socket says 172.21.15.98:49610) 2024-03-16T12:18:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.165+0000 7f8279200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82741074b0 0x7f8274198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.165+0000 7f827a600640 1 -- 172.21.15.98:0/466945713 learned_addr learned my addr 172.21.15.98:0/466945713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 -- 172.21.15.98:0/466945713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82741074b0 msgr2=0x7f8274198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82741074b0 0x7f8274198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 -- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8274104b80 msgr2=0x7f8274197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8274104b80 0x7f8274197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 -- 172.21.15.98:0/466945713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827406b330 con 0x7f827419d290 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827a600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f826c010ea0 tx=0x7f826c00b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c004b20 con 0x7f827419d290 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c004cc0 con 0x7f827419d290 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82741000a0 con 0x7f827419d290 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c019850 con 0x7f827419d290 2024-03-16T12:18:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.166+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741005c0 con 0x7f827419d290 2024-03-16T12:18:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.168+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82740fd730 con 0x7f827419d290 2024-03-16T12:18:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.171+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f826c019cc0 con 0x7f827419d290 2024-03-16T12:18:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.172+0000 7f8272000640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 0x7f825c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.172+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f826c0d2630 con 0x7f827419d290 2024-03-16T12:18:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.173+0000 7f8279c00640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 0x7f825c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.173+0000 7f8279c00640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 0x7f825c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8268009780 tx=0x7f826802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.176+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f826c09fad0 con 0x7f827419d290 2024-03-16T12:18:26.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:25 smithi098 ceph-mon[31633]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:26.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:25 smithi153 ceph-mon[36901]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:26.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:25 smithi173 ceph-mon[36632]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.448+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82741019f0 con 0x7f827419d290 2024-03-16T12:18:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.448+0000 7f8272000640 1 -- 172.21.15.98:0/466945713 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f826c09f3d0 con 0x7f827419d290 2024-03-16T12:18:26.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:26.432 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T12:18:26.432 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 msgr2=0x7f825c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 0x7f825c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8268009780 tx=0x7f826802e040 comp rx=0 tx=0).stop 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 msgr2=0x7f827419f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f826c010ea0 tx=0x7f826c00b690 comp rx=0 tx=0).stop 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 shutdown_connections 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f825c0a2800 0x7f825c0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f827419d290 0x7f827419f680 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82741074b0 0x7f8274198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 --2- 172.21.15.98:0/466945713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8274104b80 0x7f8274197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 >> 172.21.15.98:0/466945713 conn(0x7f82740fbc40 msgr2=0x7f82741040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 shutdown_connections 2024-03-16T12:18:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:26.451+0000 7f827c600640 1 -- 172.21.15.98:0/466945713 wait complete. 2024-03-16T12:18:27.111 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:26 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/466945713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:27.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:26 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/466945713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:27.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:26 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/466945713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:28.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:28.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:28.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:28.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 -- 172.21.15.98:0/1196222455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 msgr2=0x7f4688104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 --2- 172.21.15.98:0/1196222455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688104e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4670009900 tx=0x7f467002bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 -- 172.21.15.98:0/1196222455 shutdown_connections 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 --2- 172.21.15.98:0/1196222455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 0x7f468810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 --2- 172.21.15.98:0/1196222455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688104e80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 --2- 172.21.15.98:0/1196222455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 -- 172.21.15.98:0/1196222455 >> 172.21.15.98:0/1196222455 conn(0x7f46880fed80 msgr2=0x7f46881011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 -- 172.21.15.98:0/1196222455 shutdown_connections 2024-03-16T12:18:28.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.695+0000 7f468ce00640 1 -- 172.21.15.98:0/1196222455 wait complete. 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.696+0000 7f468ce00640 1 Processor -- start 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.696+0000 7f468ce00640 1 -- start start 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 0x7f46881a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f468810f4c0 con 0x7f4688104a00 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f468810f340 con 0x7f46881053c0 2024-03-16T12:18:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f468ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f468810f640 con 0x7f4688103810 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f467ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f467ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52484/0 (socket says 172.21.15.98:52484) 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52574/0 (socket says 172.21.15.98:52574) 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f467ea00640 1 -- 172.21.15.98:0/1321171535 learned_addr learned my addr 172.21.15.98:0/1321171535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687e00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 0x7f46881a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 -- 172.21.15.98:0/1321171535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 msgr2=0x7f46881a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 0x7f46881a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 -- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 msgr2=0x7f4688199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f4687400640 1 -- 172.21.15.98:0/1321171535 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46881a16c0 con 0x7f4688103810 2024-03-16T12:18:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.697+0000 7f467ea00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f4687400640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f467800d6a0 tx=0x7f467800db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678017070 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46881a19b0 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4678009d00 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678003c60 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.698+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46881a1ed0 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.699+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468810ab30 con 0x7f4688103810 2024-03-16T12:18:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.701+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4678009820 con 0x7f4688103810 2024-03-16T12:18:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.702+0000 7f4684c00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 0x7f466c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.703+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f46780cdf20 con 0x7f4688103810 2024-03-16T12:18:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.703+0000 7f467ea00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 0x7f466c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.703+0000 7f467ea00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 0x7f466c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f467002c8e0 tx=0x7f467002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.707+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f467809ad70 con 0x7f4688103810 2024-03-16T12:18:28.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.963+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f468802d1b0 con 0x7f4688103810 2024-03-16T12:18:28.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.964+0000 7f4684c00640 1 -- 172.21.15.98:0/1321171535 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f467809ee10 con 0x7f4688103810 2024-03-16T12:18:28.948 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:28.948 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T12:18:28.948 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 msgr2=0x7f466c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 0x7f466c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f467002c8e0 tx=0x7f467002df90 comp rx=0 tx=0).stop 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 msgr2=0x7f4688199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f467800d6a0 tx=0x7f467800db60 comp rx=0 tx=0).stop 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 shutdown_connections 2024-03-16T12:18:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f466c0a2800 0x7f466c0a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46881053c0 0x7f46881a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4688104a00 0x7f4688199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 --2- 172.21.15.98:0/1321171535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4688103810 0x7f4688199aa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 >> 172.21.15.98:0/1321171535 conn(0x7f46880fed80 msgr2=0x7f4688107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 shutdown_connections 2024-03-16T12:18:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:28.967+0000 7f468ce00640 1 -- 172.21.15.98:0/1321171535 wait complete. 2024-03-16T12:18:28.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:28 smithi098 ceph-mon[31633]: Deploying daemon mds.a on smithi098 2024-03-16T12:18:28.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:28 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.98:6838/1905625242,v1:172.21.15.98:6839/1905625242] up:boot 2024-03-16T12:18:28.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:28 smithi098 ceph-mon[31633]: fsmap 10 up:standby 2024-03-16T12:18:28.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T12:18:29.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:28 smithi153 ceph-mon[36901]: Deploying daemon mds.a on smithi098 2024-03-16T12:18:29.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:28 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.98:6838/1905625242,v1:172.21.15.98:6839/1905625242] up:boot 2024-03-16T12:18:29.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:28 smithi153 ceph-mon[36901]: fsmap 10 up:standby 2024-03-16T12:18:29.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T12:18:29.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:28 smithi173 ceph-mon[36632]: Deploying daemon mds.a on smithi098 2024-03-16T12:18:29.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:28 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.98:6838/1905625242,v1:172.21.15.98:6839/1905625242] up:boot 2024-03-16T12:18:29.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:28 smithi173 ceph-mon[36632]: fsmap 10 up:standby 2024-03-16T12:18:29.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T12:18:30.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:29 smithi098 ceph-mon[31633]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:30.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1321171535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:30.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:29 smithi153 ceph-mon[36901]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:30.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1321171535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:30.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:29 smithi173 ceph-mon[36632]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:30.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1321171535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:30.936 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:30 smithi098 systemd[1]: Starting Ceph mds.a for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:30.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 -- 172.21.15.98:0/3695444774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 msgr2=0x7f396410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 --2- 172.21.15.98:0/3695444774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f396410be60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f395c009ca0 tx=0x7f395c02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 -- 172.21.15.98:0/3695444774 shutdown_connections 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 --2- 172.21.15.98:0/3695444774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f396410be60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 --2- 172.21.15.98:0/3695444774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f3964105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 --2- 172.21.15.98:0/3695444774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 0x7f3964103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 -- 172.21.15.98:0/3695444774 >> 172.21.15.98:0/3695444774 conn(0x7f39640fef90 msgr2=0x7f39641013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 -- 172.21.15.98:0/3695444774 shutdown_connections 2024-03-16T12:18:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.207+0000 7f396b000640 1 -- 172.21.15.98:0/3695444774 wait complete. 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.208+0000 7f396b000640 1 Processor -- start 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.208+0000 7f396b000640 1 -- start start 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 0x7f3964199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f39641a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f396410f680 con 0x7f39641055d0 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f396410f500 con 0x7f3964103a20 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f396b000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f396410f800 con 0x7f3964104c10 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f3969000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f39641a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f3963e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 0x7f3964199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52588/0 (socket says 172.21.15.98:52588) 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 -- 172.21.15.98:0/2990968880 learned_addr learned my addr 172.21.15.98:0/2990968880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 -- 172.21.15.98:0/2990968880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 msgr2=0x7f3964199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 0x7f3964199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 -- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 msgr2=0x7f39641a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f39641a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f395b400640 1 -- 172.21.15.98:0/2990968880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39641a1890 con 0x7f3964104c10 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.209+0000 7f3969000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f39641a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f395b400640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f394c00d950 tx=0x7f394c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c017030 con 0x7f3964104c10 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f394c004020 con 0x7f3964104c10 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39641a1b80 con 0x7f3964104c10 2024-03-16T12:18:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c0047a0 con 0x7f3964104c10 2024-03-16T12:18:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.210+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39641a20d0 con 0x7f3964104c10 2024-03-16T12:18:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.211+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396410ace0 con 0x7f3964104c10 2024-03-16T12:18:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.213+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f394c0041d0 con 0x7f3964104c10 2024-03-16T12:18:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.214+0000 7f3961600640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 0x7f39480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.214+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f394c0cdfe0 con 0x7f3964104c10 2024-03-16T12:18:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.214+0000 7f3963e00640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 0x7f39480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.215+0000 7f3963e00640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 0x7f39480a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f396419b150 tx=0x7f3954005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.218+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f394c09b4e0 con 0x7f3964104c10 2024-03-16T12:18:31.244 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:30 smithi098 podman[73609]: 2024-03-16 12:18:30.859403995 +0000 UTC m=+0.038200869 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:31.244 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:30 smithi098 podman[73609]: 2024-03-16 12:18:30.991646523 +0000 UTC m=+0.170443403 container create ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T12:18:31.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.463+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f396402d1b0 con 0x7f3964104c10 2024-03-16T12:18:31.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.463+0000 7f3961600640 1 -- 172.21.15.98:0/2990968880 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f394c09ade0 con 0x7f3964104c10 2024-03-16T12:18:31.447 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:31.447 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T12:18:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 msgr2=0x7f39480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 0x7f39480a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f396419b150 tx=0x7f3954005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 msgr2=0x7f396419a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f394c00d950 tx=0x7f394c00de10 comp rx=0 tx=0).stop 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 shutdown_connections 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f39480a2800 0x7f39480a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39641055d0 0x7f39641a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3964104c10 0x7f396419a1c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 --2- 172.21.15.98:0/2990968880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3964103a20 0x7f3964199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:31.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.466+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 >> 172.21.15.98:0/2990968880 conn(0x7f39640fef90 msgr2=0x7f3964107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:31.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.467+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 shutdown_connections 2024-03-16T12:18:31.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:31.467+0000 7f396b000640 1 -- 172.21.15.98:0/2990968880 wait complete. 2024-03-16T12:18:31.744 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 podman[73609]: 2024-03-16 12:18:31.279488737 +0000 UTC m=+0.458285623 container init ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T12:18:31.745 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 podman[73609]: 2024-03-16 12:18:31.289906613 +0000 UTC m=+0.468703461 container start ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T12:18:31.745 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mds[73667]: -- 172.21.15.98:0/3199000712 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f7eb4da40 con 0x556f7eb20800 2024-03-16T12:18:31.745 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a[73644]: starting mds.a at 2024-03-16T12:18:31.745 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 bash[73609]: ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b 2024-03-16T12:18:31.745 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:18:31 smithi098 systemd[1]: Started Ceph mds.a for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:32.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2990968880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:32.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:31 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2990968880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:32.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:32.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:31 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2990968880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T12:18:32.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:31 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:33.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:32 smithi153 ceph-mon[36901]: Deploying daemon mds.b on smithi153 2024-03-16T12:18:33.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:32 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.98:6840/1343544864,v1:172.21.15.98:6841/1343544864] up:boot 2024-03-16T12:18:33.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:32 smithi153 ceph-mon[36901]: fsmap 11 up:standby 2024-03-16T12:18:33.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:32 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T12:18:33.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:32 smithi098 ceph-mon[31633]: Deploying daemon mds.b on smithi153 2024-03-16T12:18:33.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:32 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.98:6840/1343544864,v1:172.21.15.98:6841/1343544864] up:boot 2024-03-16T12:18:33.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:32 smithi098 ceph-mon[31633]: fsmap 11 up:standby 2024-03-16T12:18:33.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:32 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T12:18:33.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:32 smithi173 ceph-mon[36632]: Deploying daemon mds.b on smithi153 2024-03-16T12:18:33.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:32 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.98:6840/1343544864,v1:172.21.15.98:6841/1343544864] up:boot 2024-03-16T12:18:33.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:32 smithi173 ceph-mon[36632]: fsmap 11 up:standby 2024-03-16T12:18:33.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:32 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T12:18:33.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 -- 172.21.15.98:0/1446610310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7481099b0 msgr2=0x7fc748110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 --2- 172.21.15.98:0/1446610310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7481099b0 0x7fc748110240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc740009ca0 tx=0x7fc74002c100 comp rx=0 tx=0).stop 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 -- 172.21.15.98:0/1446610310 shutdown_connections 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 --2- 172.21.15.98:0/1446610310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7481099b0 0x7fc748110240 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 --2- 172.21.15.98:0/1446610310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc748108ff0 0x7fc748109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 --2- 172.21.15.98:0/1446610310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748107e00 0x7fc748108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 -- 172.21.15.98:0/1446610310 >> 172.21.15.98:0/1446610310 conn(0x7fc748076dc0 msgr2=0x7fc7480791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:33.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 -- 172.21.15.98:0/1446610310 shutdown_connections 2024-03-16T12:18:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.712+0000 7fc74f600640 1 -- 172.21.15.98:0/1446610310 wait complete. 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 Processor -- start 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 -- start start 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 0x7fc74819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 0x7fc7481a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc748113a80 con 0x7fc748108ff0 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc748113900 con 0x7fc748107e00 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74f600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc748113c00 con 0x7fc7481099b0 2024-03-16T12:18:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.713+0000 7fc74cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc747e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 0x7fc74819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44872/0 (socket says 172.21.15.98:44872) 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74cc00640 1 -- 172.21.15.98:0/1717412233 learned_addr learned my addr 172.21.15.98:0/1717412233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 0x7fc7481a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 -- 172.21.15.98:0/1717412233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 msgr2=0x7fc74819e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 -- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 msgr2=0x7fc74819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 0x7fc74819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 -- 172.21.15.98:0/1717412233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7481a5cb0 con 0x7fc7481099b0 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74cc00640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc747e00640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 0x7fc74819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.714+0000 7fc74d600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 0x7fc7481a5640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc74002c640 tx=0x7fc740035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.715+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740039040 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.715+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc740044070 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.715+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7481a5f40 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.715+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7400043e0 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.715+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7481a63a0 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.716+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74810f120 con 0x7fc7481099b0 2024-03-16T12:18:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.718+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc740004580 con 0x7fc7481099b0 2024-03-16T12:18:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.719+0000 7fc745600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 0x7fc7300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.719+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc7400f0c70 con 0x7fc7481099b0 2024-03-16T12:18:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.719+0000 7fc74cc00640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 0x7fc7300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.720+0000 7fc74cc00640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 0x7fc7300a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0058c0 tx=0x7fc73c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.723+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7400be110 con 0x7fc7481099b0 2024-03-16T12:18:33.886 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:33 smithi153 systemd[1]: Starting Ceph mds.b for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:18:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.964+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc748062480 con 0x7fc7481099b0 2024-03-16T12:18:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.965+0000 7fc745600640 1 -- 172.21.15.98:0/1717412233 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fc7400bda10 con 0x7fc7481099b0 2024-03-16T12:18:33.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:33.949 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T12:18:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 msgr2=0x7fc7300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 0x7fc7300a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0058c0 tx=0x7fc73c005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 msgr2=0x7fc7481a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 0x7fc7481a5640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc74002c640 tx=0x7fc740035d90 comp rx=0 tx=0).stop 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 shutdown_connections 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc7300a2800 0x7fc7300a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7481099b0 0x7fc7481a5640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc748108ff0 0x7fc74819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 --2- 172.21.15.98:0/1717412233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc748107e00 0x7fc74819e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 >> 172.21.15.98:0/1717412233 conn(0x7fc748076dc0 msgr2=0x7fc74810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 shutdown_connections 2024-03-16T12:18:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:33.968+0000 7fc74f600640 1 -- 172.21.15.98:0/1717412233 wait complete. 2024-03-16T12:18:34.141 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:33 smithi153 ceph-mon[36901]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:34.142 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:33 smithi153 podman[67640]: 2024-03-16 12:18:33.895606666 +0000 UTC m=+0.018120685 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:18:34.142 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 podman[67640]: 2024-03-16 12:18:34.037268485 +0000 UTC m=+0.159782536 container create 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T12:18:34.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:33 smithi098 ceph-mon[31633]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:34.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:33 smithi173 ceph-mon[36632]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:34.405 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 podman[67640]: 2024-03-16 12:18:34.323403256 +0000 UTC m=+0.445917262 container init 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-16T12:18:34.405 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 podman[67640]: 2024-03-16 12:18:34.32675467 +0000 UTC m=+0.449268672 container start 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T12:18:34.406 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mds[67659]: -- 172.21.15.153:0/2291065495 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565210041a40 con 0x565210014800 2024-03-16T12:18:34.406 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: starting mds.b at 2024-03-16T12:18:34.755 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 bash[67640]: 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 2024-03-16T12:18:34.755 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:34 smithi153 systemd[1]: Started Ceph mds.b for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1717412233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:34 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1717412233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:34 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1717412233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:35.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:34 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:35.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:35 smithi153 ceph-mon[36901]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:35.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:35 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:boot 2024-03-16T12:18:35.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:35 smithi153 ceph-mon[36901]: fsmap 12 up:standby 2024-03-16T12:18:35.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:35 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:18:35.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:36.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:35 smithi173 ceph-mon[36632]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:36.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:35 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:boot 2024-03-16T12:18:36.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:35 smithi173 ceph-mon[36632]: fsmap 12 up:standby 2024-03-16T12:18:36.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:35 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:18:36.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 -- 172.21.15.98:0/2911185565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260103a00 msgr2=0x7f3260103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 --2- 172.21.15.98:0/2911185565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260103a00 0x7f3260103e20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3254009900 tx=0x7f325402bc50 comp rx=0 tx=0).stop 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 -- 172.21.15.98:0/2911185565 shutdown_connections 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 --2- 172.21.15.98:0/2911185565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32601055b0 0x7f326010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 --2- 172.21.15.98:0/2911185565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f3260105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 --2- 172.21.15.98:0/2911185565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3260103a00 0x7f3260103e20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 -- 172.21.15.98:0/2911185565 >> 172.21.15.98:0/2911185565 conn(0x7f32600fef90 msgr2=0x7f32601013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 -- 172.21.15.98:0/2911185565 shutdown_connections 2024-03-16T12:18:36.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.203+0000 7f3267600640 1 -- 172.21.15.98:0/2911185565 wait complete. 2024-03-16T12:18:36.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.204+0000 7f3267600640 1 Processor -- start 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.204+0000 7f3267600640 1 -- start start 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3260103a00 0x7f3260199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f326019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f650 con 0x7f32601055b0 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f4d0 con 0x7f3260104bf0 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3267600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f7d0 con 0x7f3260103a00 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f325fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f326019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52550/0 (socket says 172.21.15.98:52550) 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 -- 172.21.15.98:0/3697257509 learned_addr learned my addr 172.21.15.98:0/3697257509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3264c00640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3260103a00 0x7f3260199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 -- 172.21.15.98:0/3697257509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3260103a00 msgr2=0x7f3260199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3260103a00 0x7f3260199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 -- 172.21.15.98:0/3697257509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 msgr2=0x7f326019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f326019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f3265600640 1 -- 172.21.15.98:0/3697257509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601a19d0 con 0x7f32601055b0 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.205+0000 7f325fe00640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f326019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f3265600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f325800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258004b20 con 0x7f32601055b0 2024-03-16T12:18:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3258004cc0 con 0x7f32601055b0 2024-03-16T12:18:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601a1cc0 con 0x7f32601055b0 2024-03-16T12:18:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258010760 con 0x7f32601055b0 2024-03-16T12:18:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.206+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601a2190 con 0x7f32601055b0 2024-03-16T12:18:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.207+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326010ada0 con 0x7f32601055b0 2024-03-16T12:18:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.209+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3258005020 con 0x7f32601055b0 2024-03-16T12:18:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.210+0000 7f325d600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 0x7f32480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.210+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f32580cec60 con 0x7f32601055b0 2024-03-16T12:18:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.211+0000 7f3264c00640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 0x7f32480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.211+0000 7f3264c00640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 0x7f32480a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3254000c00 tx=0x7f3254005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.215+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f325809c100 con 0x7f32601055b0 2024-03-16T12:18:36.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:35 smithi098 ceph-mon[31633]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:35 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:boot 2024-03-16T12:18:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:35 smithi098 ceph-mon[31633]: fsmap 12 up:standby 2024-03-16T12:18:36.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:35 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:18:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.460+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f326002d1b0 con 0x7f32601055b0 2024-03-16T12:18:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.461+0000 7f325d600640 1 -- 172.21.15.98:0/3697257509 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7f325809ba00 con 0x7f32601055b0 2024-03-16T12:18:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:36.445 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-16T12:18:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-03-16T12:18:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 msgr2=0x7f32480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 0x7f32480a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3254000c00 tx=0x7f3254005fd0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 msgr2=0x7f32601a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f325800c2d0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 shutdown_connections 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f32480a2800 0x7f32480a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32601055b0 0x7f32601a1330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3260104bf0 0x7f326019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 --2- 172.21.15.98:0/3697257509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3260103a00 0x7f3260199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 >> 172.21.15.98:0/3697257509 conn(0x7f32600fef90 msgr2=0x7f3260107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 shutdown_connections 2024-03-16T12:18:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.464+0000 7f3267600640 1 -- 172.21.15.98:0/3697257509 wait complete. 2024-03-16T12:18:36.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 -- 172.21.15.98:0/1848132536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01013f0 msgr2=0x7feee0101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 --2- 172.21.15.98:0/1848132536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01013f0 0x7feee0101870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feed8009ca0 tx=0x7feed802c190 comp rx=0 tx=0).stop 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 -- 172.21.15.98:0/1848132536 shutdown_connections 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 --2- 172.21.15.98:0/1848132536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01013f0 0x7feee0101870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 --2- 172.21.15.98:0/1848132536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 --2- 172.21.15.98:0/1848132536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01000f0 0x7feee0100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.708+0000 7feee6c00640 1 -- 172.21.15.98:0/1848132536 >> 172.21.15.98:0/1848132536 conn(0x7feee00fbc70 msgr2=0x7feee00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.709+0000 7feee6c00640 1 -- 172.21.15.98:0/1848132536 shutdown_connections 2024-03-16T12:18:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.709+0000 7feee6c00640 1 -- 172.21.15.98:0/1848132536 wait complete. 2024-03-16T12:18:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 Processor -- start 2024-03-16T12:18:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 -- start start 2024-03-16T12:18:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 0x7feee0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 0x7feee01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feee010f680 con 0x7feee01013f0 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feee010f500 con 0x7feee0100a50 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee6c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feee010f800 con 0x7feee01000f0 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feedf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feee4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 0x7feee01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feedfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 0x7feee0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feedf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44904/0 (socket says 172.21.15.98:44904) 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.710+0000 7feedf400640 1 -- 172.21.15.98:0/2220855841 learned_addr learned my addr 172.21.15.98:0/2220855841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 -- 172.21.15.98:0/2220855841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 msgr2=0x7feee019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 -- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 msgr2=0x7feee01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 0x7feee01a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 -- 172.21.15.98:0/2220855841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee01a18d0 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feee4c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 0x7feee01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedfe00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 0x7feee0199c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feed0005fd0 tx=0x7feed000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0009c30 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed0009dd0 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee01a1bc0 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.711+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0010460 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.712+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee01a2160 con 0x7feee01000f0 2024-03-16T12:18:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.713+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee010af80 con 0x7feee01000f0 2024-03-16T12:18:36.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.714+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7feed0003ef0 con 0x7feee01000f0 2024-03-16T12:18:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.715+0000 7feedcc00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 0x7feec40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.716+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feed00ce100 con 0x7feee01000f0 2024-03-16T12:18:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.716+0000 7feedf400640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 0x7feec40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.717+0000 7feedf400640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 0x7feec40a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feee019b190 tx=0x7feecc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.720+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feed0017020 con 0x7feee01000f0 2024-03-16T12:18:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.962+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feee00623e0 con 0x7feee01000f0 2024-03-16T12:18:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.963+0000 7feedcc00640 1 -- 172.21.15.98:0/2220855841 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7feed009af50 con 0x7feee01000f0 2024-03-16T12:18:36.946 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:36.946 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:17:44.481936+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:18:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 msgr2=0x7feec40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 0x7feec40a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feee019b190 tx=0x7feecc005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 msgr2=0x7feee0199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 0x7feee0199c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feed0005fd0 tx=0x7feed000dde0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 shutdown_connections 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7feec40a2880 0x7feec40a4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee01013f0 0x7feee01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feee0100a50 0x7feee019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 --2- 172.21.15.98:0/2220855841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feee01000f0 0x7feee0199c90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 >> 172.21.15.98:0/2220855841 conn(0x7feee00fbc70 msgr2=0x7feee00fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 shutdown_connections 2024-03-16T12:18:36.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:36.966+0000 7feee6c00640 1 -- 172.21.15.98:0/2220855841 wait complete. 2024-03-16T12:18:36.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T12:18:36.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:18:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.174+0000 7fe732c00640 1 -- 172.21.15.98:0/4007433119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 msgr2=0x7fe72c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.174+0000 7fe732c00640 1 --2- 172.21.15.98:0/4007433119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c101c10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009900 tx=0x7fe71c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 -- 172.21.15.98:0/4007433119 shutdown_connections 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 --2- 172.21.15.98:0/4007433119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1033a0 0x7fe72c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 --2- 172.21.15.98:0/4007433119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1029e0 0x7fe72c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 --2- 172.21.15.98:0/4007433119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c101c10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 -- 172.21.15.98:0/4007433119 >> 172.21.15.98:0/4007433119 conn(0x7fe72c0fcd60 msgr2=0x7fe72c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 -- 172.21.15.98:0/4007433119 shutdown_connections 2024-03-16T12:18:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.175+0000 7fe732c00640 1 -- 172.21.15.98:0/4007433119 wait complete. 2024-03-16T12:18:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 Processor -- start 2024-03-16T12:18:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 -- start start 2024-03-16T12:18:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 0x7fe72c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 0x7fe72c1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10d490 con 0x7fe72c1033a0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10d310 con 0x7fe72c1017f0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe732c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10d610 con 0x7fe72c1029e0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe72be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe72be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33276/0 (socket says 172.21.15.98:33276) 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.176+0000 7fe72be00640 1 -- 172.21.15.98:0/1600698643 learned_addr learned my addr 172.21.15.98:0/1600698643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe730c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 0x7fe72c1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72b400640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 0x7fe72c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 -- 172.21.15.98:0/1600698643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 msgr2=0x7fe72c19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 0x7fe72c19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 -- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 msgr2=0x7fe72c1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 0x7fe72c1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 -- 172.21.15.98:0/1600698643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c1a1960 con 0x7fe72c1017f0 2024-03-16T12:18:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72b400640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 0x7fe72c19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe730c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 0x7fe72c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe72be00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe71c000c00 tx=0x7fe71c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c043020 con 0x7fe72c1017f0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe71c0054e0 con 0x7fe72c1017f0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c003c40 con 0x7fe72c1017f0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.177+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72c1a1bf0 con 0x7fe72c1017f0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.178+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72c1a20f0 con 0x7fe72c1017f0 2024-03-16T12:18:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.179+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe72c1a22c0 con 0x7fe72c1017f0 2024-03-16T12:18:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.180+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe71c046020 con 0x7fe72c1017f0 2024-03-16T12:18:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.182+0000 7fe728c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 0x7fe7100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.182+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe71c0f07a0 con 0x7fe72c1017f0 2024-03-16T12:18:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.182+0000 7fe72b400640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 0x7fe7100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.183+0000 7fe72b400640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 0x7fe7100a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19b250 tx=0x7fe718005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.186+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe71c04d020 con 0x7fe72c1017f0 2024-03-16T12:18:37.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:36 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3697257509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:37.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:36 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:36 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3697257509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:37.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:36 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3697257509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:37.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:36 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:37.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.424+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe72c1a2650 con 0x7fe72c1017f0 2024-03-16T12:18:37.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.425+0000 7fe728c00640 1 -- 172.21.15.98:0/1600698643 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fe71c03c350 con 0x7fe72c1017f0 2024-03-16T12:18:37.408 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:18:37.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 msgr2=0x7fe7100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 0x7fe7100a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19b250 tx=0x7fe718005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:37.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 msgr2=0x7fe72c199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe71c000c00 tx=0x7fe71c02c650 comp rx=0 tx=0).stop 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 shutdown_connections 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe7100a2800 0x7fe7100a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe72c1033a0 0x7fe72c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe72c1029e0 0x7fe72c19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 --2- 172.21.15.98:0/1600698643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1017f0 0x7fe72c199d40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 >> 172.21.15.98:0/1600698643 conn(0x7fe72c0fcd60 msgr2=0x7fe72c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.427+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 shutdown_connections 2024-03-16T12:18:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.428+0000 7fe732c00640 1 -- 172.21.15.98:0/1600698643 wait complete. 2024-03-16T12:18:37.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T12:18:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 -- 172.21.15.98:0/3994145704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010100530 msgr2=0x7f0010100950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 --2- 172.21.15.98:0/3994145704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010100530 0x7f0010100950 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0000009900 tx=0x7f000002bc50 comp rx=0 tx=0).stop 2024-03-16T12:18:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 -- 172.21.15.98:0/3994145704 shutdown_connections 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 --2- 172.21.15.98:0/3994145704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00101020e0 0x7f0010108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 --2- 172.21.15.98:0/3994145704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010101720 0x7f0010101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 --2- 172.21.15.98:0/3994145704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010100530 0x7f0010100950 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.652+0000 7f000fe00640 1 -- 172.21.15.98:0/3994145704 >> 172.21.15.98:0/3994145704 conn(0x7f00100fbac0 msgr2=0x7f00100fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.653+0000 7f000fe00640 1 -- 172.21.15.98:0/3994145704 shutdown_connections 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.653+0000 7f000fe00640 1 -- 172.21.15.98:0/3994145704 wait complete. 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.653+0000 7f000fe00640 1 Processor -- start 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 -- start start 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 0x7f0010199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 0x7f001019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 0x7f00101a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f001010c1c0 con 0x7f0010100530 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f001010c040 con 0x7f00101020e0 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f001010c340 con 0x7f0010101720 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 0x7f0010199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 0x7f00101a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 0x7f001019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 0x7f0010199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52506/0 (socket says 172.21.15.98:52506) 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.654+0000 7f000ea00640 1 -- 172.21.15.98:0/1020361098 learned_addr learned my addr 172.21.15.98:0/1020361098 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 -- 172.21.15.98:0/1020361098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 msgr2=0x7f001019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 0x7f001019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 -- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 msgr2=0x7f0010199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 0x7f0010199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 -- 172.21.15.98:0/1020361098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00101a1860 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000e000640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 0x7f001019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000f400640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 0x7f00101a11c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0008009fd0 tx=0x7f000800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000800ce20 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00101a1b50 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0008005020 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008010720 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.655+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00101a20f0 con 0x7f00101020e0 2024-03-16T12:18:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.656+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00101a22c0 con 0x7f00101020e0 2024-03-16T12:18:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.660+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0008005540 con 0x7f00101020e0 2024-03-16T12:18:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.661+0000 7f0007400640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 0x7efff00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.662+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f00080ceda0 con 0x7f00101020e0 2024-03-16T12:18:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.662+0000 7f000ea00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 0x7efff00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.662+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f000809c240 con 0x7f00101020e0 2024-03-16T12:18:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.663+0000 7f000ea00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 0x7efff00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0000006010 tx=0x7f0000005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.867+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f00101a2490 con 0x7f00101020e0 2024-03-16T12:18:37.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.868+0000 7f0007400640 1 -- 172.21.15.98:0/1020361098 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f000809bb40 con 0x7f00101020e0 2024-03-16T12:18:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:18:37.852 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 msgr2=0x7efff00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 0x7efff00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0000006010 tx=0x7f0000005b50 comp rx=0 tx=0).stop 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 msgr2=0x7f00101a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 0x7f00101a11c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0008009fd0 tx=0x7f000800ef10 comp rx=0 tx=0).stop 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 shutdown_connections 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff00a2800 0x7efff00a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00101020e0 0x7f00101a11c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010101720 0x7f001019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 --2- 172.21.15.98:0/1020361098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0010100530 0x7f0010199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 >> 172.21.15.98:0/1020361098 conn(0x7f00100fbac0 msgr2=0x7f00100fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 shutdown_connections 2024-03-16T12:18:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:37.871+0000 7f000fe00640 1 -- 172.21.15.98:0/1020361098 wait complete. 2024-03-16T12:18:37.875 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T12:18:37.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 -- 172.21.15.98:0/844414985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401028d0 msgr2=0x7fde40102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 --2- 172.21.15.98:0/844414985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401028d0 0x7fde40102d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde28009860 tx=0x7fde2802bb70 comp rx=0 tx=0).stop 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 -- 172.21.15.98:0/844414985 shutdown_connections 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 --2- 172.21.15.98:0/844414985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde40103290 0x7fde40109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 --2- 172.21.15.98:0/844414985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401028d0 0x7fde40102d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 --2- 172.21.15.98:0/844414985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde401016e0 0x7fde40101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 -- 172.21.15.98:0/844414985 >> 172.21.15.98:0/844414985 conn(0x7fde400fcc50 msgr2=0x7fde400ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 -- 172.21.15.98:0/844414985 shutdown_connections 2024-03-16T12:18:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.082+0000 7fde45600640 1 -- 172.21.15.98:0/844414985 wait complete. 2024-03-16T12:18:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.083+0000 7fde45600640 1 Processor -- start 2024-03-16T12:18:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.083+0000 7fde45600640 1 -- start start 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde401028d0 0x7fde4019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde40103290 0x7fde401a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010d360 con 0x7fde401028d0 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010d1e0 con 0x7fde40103290 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde45600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010d4e0 con 0x7fde401016e0 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde44c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde40103290 0x7fde401a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38722/0 (socket says 172.21.15.98:38722) 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 -- 172.21.15.98:0/2638751983 learned_addr learned my addr 172.21.15.98:0/2638751983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde37400640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde401028d0 0x7fde4019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 -- 172.21.15.98:0/2638751983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde40103290 msgr2=0x7fde401a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde40103290 0x7fde401a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 -- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde401028d0 msgr2=0x7fde4019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde401028d0 0x7fde4019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.084+0000 7fde3fe00640 1 -- 172.21.15.98:0/2638751983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde401a18b0 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde3fe00640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde3000eb70 tx=0x7fde3000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30003b70 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde3000ee70 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30010830 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde401a1ba0 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.085+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde401a2070 con 0x7fde401016e0 2024-03-16T12:18:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.086+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde401a2240 con 0x7fde401016e0 2024-03-16T12:18:38.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.088+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fde30004a40 con 0x7fde401016e0 2024-03-16T12:18:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.089+0000 7fde3d600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 0x7fde240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.089+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fde300cec80 con 0x7fde401016e0 2024-03-16T12:18:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.089+0000 7fde37400640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 0x7fde240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.090+0000 7fde37400640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 0x7fde240a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde28009830 tx=0x7fde28034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.095+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde3009c120 con 0x7fde401016e0 2024-03-16T12:18:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2220855841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1600698643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1020361098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T12:18:38.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:37 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2220855841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1600698643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1020361098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T12:18:38.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:37 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.283+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fde401a2630 con 0x7fde401016e0 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2220855841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1600698643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1020361098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T12:18:38.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:37 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:18:38.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.937+0000 7fde3d600640 1 -- 172.21.15.98:0/2638751983 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fde3009ba20 con 0x7fde401016e0 2024-03-16T12:18:38.922 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 msgr2=0x7fde240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 0x7fde240a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde28009830 tx=0x7fde28034ab0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 msgr2=0x7fde40199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde3000eb70 tx=0x7fde3000c6a0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 shutdown_connections 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fde240a2800 0x7fde240a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde40103290 0x7fde401a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde401028d0 0x7fde4019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 --2- 172.21.15.98:0/2638751983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde401016e0 0x7fde40199c70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:38.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.944+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 >> 172.21.15.98:0/2638751983 conn(0x7fde400fcc50 msgr2=0x7fde40073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:38.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.945+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 shutdown_connections 2024-03-16T12:18:38.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:38.945+0000 7fde45600640 1 -- 172.21.15.98:0/2638751983 wait complete. 2024-03-16T12:18:38.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-16T12:18:39.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:38 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2638751983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:38 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.159+0000 7f445d200640 1 -- 172.21.15.98:0/9355627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 msgr2=0x7f4458101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.159+0000 7f445d200640 1 --2- 172.21.15.98:0/9355627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458101c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4448009900 tx=0x7f444802bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 -- 172.21.15.98:0/9355627 shutdown_connections 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 --2- 172.21.15.98:0/9355627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 0x7f4458109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 --2- 172.21.15.98:0/9355627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 0x7f4458102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 --2- 172.21.15.98:0/9355627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458101c10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 -- 172.21.15.98:0/9355627 >> 172.21.15.98:0/9355627 conn(0x7f44580fcd80 msgr2=0x7f44580ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 -- 172.21.15.98:0/9355627 shutdown_connections 2024-03-16T12:18:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.160+0000 7f445d200640 1 -- 172.21.15.98:0/9355627 wait complete. 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 Processor -- start 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 -- start start 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 0x7f4458073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 0x7f4458074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d4c0 con 0x7f44581029e0 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d340 con 0x7f44581017f0 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f445d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d640 con 0x7f44581033a0 2024-03-16T12:18:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f4455e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f4455e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33336/0 (socket says 172.21.15.98:33336) 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f4455e00640 1 -- 172.21.15.98:0/1713834193 learned_addr learned my addr 172.21.15.98:0/1713834193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f444fe00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 0x7f4458073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.161+0000 7f4456800640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 0x7f4458074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 -- 172.21.15.98:0/1713834193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 msgr2=0x7f4458074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 0x7f4458074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 -- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 msgr2=0x7f4458073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 0x7f4458073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 -- 172.21.15.98:0/1713834193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44580749a0 con 0x7f44581017f0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f4455e00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4448000c00 tx=0x7f444802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448043020 con 0x7f44581017f0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44581a1c50 con 0x7f44581017f0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44480054e0 con 0x7f44581017f0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448003c40 con 0x7f44581017f0 2024-03-16T12:18:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.162+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44581a21a0 con 0x7f44581017f0 2024-03-16T12:18:39.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.163+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4458108b60 con 0x7f44581017f0 2024-03-16T12:18:39.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.165+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4448046020 con 0x7f44581017f0 2024-03-16T12:18:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.166+0000 7f444ea00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.167+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f44480f0bd0 con 0x7f44581017f0 2024-03-16T12:18:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.167+0000 7f444fe00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:39.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.167+0000 7f444fe00640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 0x7f443c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4458075300 tx=0x7f4440005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:39.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.169+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f444804d020 con 0x7f44581017f0 2024-03-16T12:18:39.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:38 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2638751983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:38 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:38 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2638751983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:38 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T12:18:39.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.371+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f445802d1b0 con 0x7f44581017f0 2024-03-16T12:18:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.968+0000 7f444ea00640 1 -- 172.21.15.98:0/1713834193 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f44480bdf00 con 0x7f44581017f0 2024-03-16T12:18:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-03-16T12:18:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.973+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 msgr2=0x7f443c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.973+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 0x7f443c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4458075300 tx=0x7f4440005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.973+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 msgr2=0x7f4458075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.973+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4448000c00 tx=0x7f444802c650 comp rx=0 tx=0).stop 2024-03-16T12:18:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 shutdown_connections 2024-03-16T12:18:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f443c0a2800 0x7f443c0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44581033a0 0x7f4458074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44581029e0 0x7f4458073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 --2- 172.21.15.98:0/1713834193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44581017f0 0x7f4458075590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.974+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 >> 172.21.15.98:0/1713834193 conn(0x7f44580fcd80 msgr2=0x7f4458105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.975+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 shutdown_connections 2024-03-16T12:18:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:39.978+0000 7f445d200640 1 -- 172.21.15.98:0/1713834193 wait complete. 2024-03-16T12:18:39.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 -- 172.21.15.98:0/1766390912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 msgr2=0x7f205c103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 --2- 172.21.15.98:0/1766390912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c103e40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f204c009900 tx=0x7f204c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 -- 172.21.15.98:0/1766390912 shutdown_connections 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 --2- 172.21.15.98:0/1766390912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c1055d0 0x7f205c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 --2- 172.21.15.98:0/1766390912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c104c10 0x7f205c105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:40.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 --2- 172.21.15.98:0/1766390912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c103e40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:40.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 -- 172.21.15.98:0/1766390912 >> 172.21.15.98:0/1766390912 conn(0x7f205c0fef90 msgr2=0x7f205c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:40.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 -- 172.21.15.98:0/1766390912 shutdown_connections 2024-03-16T12:18:40.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.196+0000 7f2060a00640 1 -- 172.21.15.98:0/1766390912 wait complete. 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 Processor -- start 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 -- start start 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c104c10 0x7f205c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f205c10f6e0 con 0x7f205c103a20 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.197+0000 7f2060a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f205c10f560 con 0x7f205c104c10 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f2060a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f205c10f860 con 0x7f205c1055d0 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f2059e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38768/0 (socket says 172.21.15.98:38768) 2024-03-16T12:18:40.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 -- 172.21.15.98:0/1969935322 learned_addr learned my addr 172.21.15.98:0/1969935322 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f2059400640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c104c10 0x7f205c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 -- 172.21.15.98:0/1969935322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c104c10 msgr2=0x7f205c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c104c10 0x7f205c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 -- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 msgr2=0x7f205c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 -- 172.21.15.98:0/1969935322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c1a19b0 con 0x7f205c1055d0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.198+0000 7f205a800640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f205400eb10 tx=0x7f205400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.199+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054004b20 con 0x7f205c1055d0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.199+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2054004cc0 con 0x7f205c1055d0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.199+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054010760 con 0x7f205c1055d0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.199+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205c1a1ca0 con 0x7f205c1055d0 2024-03-16T12:18:40.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.199+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205c1a2170 con 0x7f205c1055d0 2024-03-16T12:18:40.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.200+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c10ad40 con 0x7f205c1055d0 2024-03-16T12:18:40.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.202+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2054005020 con 0x7f205c1055d0 2024-03-16T12:18:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.203+0000 7f2052a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.203+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f20540cef20 con 0x7f205c1055d0 2024-03-16T12:18:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.203+0000 7f2059e00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.204+0000 7f2059e00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 0x7f20400a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f204c02c170 tx=0x7f204c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.207+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f205409c0d0 con 0x7f205c1055d0 2024-03-16T12:18:40.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:39 smithi098 ceph-mon[31633]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:40.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:39 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T12:18:40.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:39 smithi098 ceph-mon[31633]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T12:18:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:39 smithi153 ceph-mon[36901]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:39 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T12:18:40.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:39 smithi153 ceph-mon[36901]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T12:18:40.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:39 smithi173 ceph-mon[36632]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:40.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:39 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T12:18:40.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:39 smithi173 ceph-mon[36632]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T12:18:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:40.451+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f205c0623e0 con 0x7f205c1055d0 2024-03-16T12:18:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.025+0000 7f2052a00640 1 -- 172.21.15.98:0/1969935322 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f205409b9d0 con 0x7f205c1055d0 2024-03-16T12:18:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T12:18:41.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.030+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 msgr2=0x7f20400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.030+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 0x7f20400a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f204c02c170 tx=0x7f204c02df90 comp rx=0 tx=0).stop 2024-03-16T12:18:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.030+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 msgr2=0x7f205c1a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.030+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f205400eb10 tx=0x7f205400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 shutdown_connections 2024-03-16T12:18:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f20400a2880 0x7f20400a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f205c1055d0 0x7f205c1a1310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f205c104c10 0x7f205c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 --2- 172.21.15.98:0/1969935322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f205c103a20 0x7f205c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.032+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 >> 172.21.15.98:0/1969935322 conn(0x7f205c0fef90 msgr2=0x7f205c107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.033+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 shutdown_connections 2024-03-16T12:18:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.034+0000 7f2060a00640 1 -- 172.21.15.98:0/1969935322 wait complete. 2024-03-16T12:18:41.039 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-16T12:18:41.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 -- 172.21.15.98:0/1792825076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4108ff0 msgr2=0x7f0be4109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 --2- 172.21.15.98:0/1792825076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4108ff0 0x7f0be4109470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009900 tx=0x7f0bd802bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 -- 172.21.15.98:0/1792825076 shutdown_connections 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 --2- 172.21.15.98:0/1792825076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be41099b0 0x7f0be4110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 --2- 172.21.15.98:0/1792825076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4108ff0 0x7f0be4109470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 --2- 172.21.15.98:0/1792825076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be4107e00 0x7f0be4108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.248+0000 7f0bec800640 1 -- 172.21.15.98:0/1792825076 >> 172.21.15.98:0/1792825076 conn(0x7f0be4076da0 msgr2=0x7f0be40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.249+0000 7f0bec800640 1 -- 172.21.15.98:0/1792825076 shutdown_connections 2024-03-16T12:18:41.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.249+0000 7f0bec800640 1 -- 172.21.15.98:0/1792825076 wait complete. 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.249+0000 7f0bec800640 1 Processor -- start 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 -- start start 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be4108ff0 0x7f0be419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be41099b0 0x7f0be41a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0be4113a80 con 0x7f0be4107e00 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0be4113900 con 0x7f0be4108ff0 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bec800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0be4113c00 con 0x7f0be41099b0 2024-03-16T12:18:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0be9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0be9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52594/0 (socket says 172.21.15.98:52594) 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0be9e00640 1 -- 172.21.15.98:0/1886900236 learned_addr learned my addr 172.21.15.98:0/1886900236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0bea800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be41099b0 0x7f0be41a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.250+0000 7f0be9400640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be4108ff0 0x7f0be419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 -- 172.21.15.98:0/1886900236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be41099b0 msgr2=0x7f0be41a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be41099b0 0x7f0be41a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 -- 172.21.15.98:0/1886900236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be4108ff0 msgr2=0x7f0be419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be4108ff0 0x7f0be419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 -- 172.21.15.98:0/1886900236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be41a5d70 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0be9e00640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000e9c0 tx=0x7f0bd000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000ccc0 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bd00035f0 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000ce60 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be41a6060 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.251+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be41a6530 con 0x7f0be4107e00 2024-03-16T12:18:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.252+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be410f1f0 con 0x7f0be4107e00 2024-03-16T12:18:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.255+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0bd00037a0 con 0x7f0be4107e00 2024-03-16T12:18:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.256+0000 7f0bdea00640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 0x7f0bcc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.256+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f0bd00ced70 con 0x7f0be4107e00 2024-03-16T12:18:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.256+0000 7f0be9400640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 0x7f0bcc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.257+0000 7f0be9400640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 0x7f0bcc0a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0bd802c7b0 tx=0x7f0bd8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.259+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bd009bec0 con 0x7f0be4107e00 2024-03-16T12:18:41.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:40 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1713834193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:40 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:40 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T12:18:41.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:40 smithi098 ceph-mon[31633]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T12:18:41.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1969935322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:41.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:41 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1713834193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1969935322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:41.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1713834193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1969935322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T12:18:41.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:18:41.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:18:41.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.466+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f0be402d1b0 con 0x7f0be4107e00 2024-03-16T12:18:41.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.484+0000 7f0bdea00640 1 -- 172.21.15.98:0/1886900236 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f0bd009b7c0 con 0x7f0be4107e00 2024-03-16T12:18:41.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 msgr2=0x7f0bcc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 0x7f0bcc0a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0bd802c7b0 tx=0x7f0bd8005fd0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 msgr2=0x7f0be419e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000e9c0 tx=0x7f0bd000edc0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 shutdown_connections 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bcc0a2880 0x7f0bcc0a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0be41099b0 0x7f0be41a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0be4108ff0 0x7f0be419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 --2- 172.21.15.98:0/1886900236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0be4107e00 0x7f0be419e150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.488+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 >> 172.21.15.98:0/1886900236 conn(0x7f0be4076da0 msgr2=0x7f0be410c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.489+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 shutdown_connections 2024-03-16T12:18:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.489+0000 7f0bec800640 1 -- 172.21.15.98:0/1886900236 wait complete. 2024-03-16T12:18:41.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T12:18:41.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 -- 172.21.15.98:0/2757571294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044107e20 msgr2=0x7f4044108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 --2- 172.21.15.98:0/2757571294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044107e20 0x7f4044108240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4034009900 tx=0x7f403402bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 -- 172.21.15.98:0/2757571294 shutdown_connections 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 --2- 172.21.15.98:0/2757571294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 0x7f4044110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 --2- 172.21.15.98:0/2757571294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044108f70 0x7f40441093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 --2- 172.21.15.98:0/2757571294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044107e20 0x7f4044108240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.733+0000 7f4048e00640 1 -- 172.21.15.98:0/2757571294 >> 172.21.15.98:0/2757571294 conn(0x7f4044076df0 msgr2=0x7f4044079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.734+0000 7f4048e00640 1 -- 172.21.15.98:0/2757571294 shutdown_connections 2024-03-16T12:18:41.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.734+0000 7f4048e00640 1 -- 172.21.15.98:0/2757571294 wait complete. 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.734+0000 7f4048e00640 1 Processor -- start 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 -- start start 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 0x7f404419e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 0x7f40441a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4044113b10 con 0x7f4044108f70 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4044113990 con 0x7f4044109930 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4048e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4044113c90 con 0x7f4044107e20 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f403be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4041e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 0x7f404419e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f403be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52604/0 (socket says 172.21.15.98:52604) 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f403be00640 1 -- 172.21.15.98:0/1598940872 learned_addr learned my addr 172.21.15.98:0/1598940872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:41.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.735+0000 7f4042800640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 0x7f40441a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 -- 172.21.15.98:0/1598940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 msgr2=0x7f40441a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 0x7f40441a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 -- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 msgr2=0x7f404419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 -- 172.21.15.98:0/1598940872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40441a5d40 con 0x7f4044107e20 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f403be00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4041e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 0x7f404419e120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4034036040 tx=0x7f403402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034043020 con 0x7f4044107e20 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40340054e0 con 0x7f4044107e20 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.736+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40441a5fd0 con 0x7f4044107e20 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.737+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034003c30 con 0x7f4044107e20 2024-03-16T12:18:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.737+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40441a6430 con 0x7f4044107e20 2024-03-16T12:18:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.738+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f404410f0d0 con 0x7f4044107e20 2024-03-16T12:18:41.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.740+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4034046020 con 0x7f4044107e20 2024-03-16T12:18:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.741+0000 7f403aa00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 0x7f40280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.741+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f40340f10a0 con 0x7f4044107e20 2024-03-16T12:18:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.741+0000 7f403be00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 0x7f40280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.742+0000 7f403be00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 0x7f40280a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f404419f630 tx=0x7f402c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:41.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.743+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f403404d020 con 0x7f4044107e20 2024-03-16T12:18:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:41.948+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f404402d1b0 con 0x7f4044107e20 2024-03-16T12:18:42.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: pgmap v313: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:42.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1598940872' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:42.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:42 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:42.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: pgmap v313: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1598940872' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:42.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:42 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: pgmap v313: 97 pgs: 12 creating+peering, 75 unknown, 10 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1886900236' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1598940872' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:42.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:42 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T12:18:43.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:43 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:active 2024-03-16T12:18:43.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:43 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:43 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:43 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:active 2024-03-16T12:18:43.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:43 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:43 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:43 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] up:active 2024-03-16T12:18:43.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:43 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:43 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:43.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.549+0000 7f403aa00640 1 -- 172.21.15.98:0/1598940872 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f40340be110 con 0x7f4044107e20 2024-03-16T12:18:43.534 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' created 2024-03-16T12:18:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 msgr2=0x7f40280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 0x7f40280a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f404419f630 tx=0x7f402c005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 msgr2=0x7f404419e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:43.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 0x7f404419e120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4034036040 tx=0x7f403402c650 comp rx=0 tx=0).stop 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 shutdown_connections 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f40280a2800 0x7f40280a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4044109930 0x7f40441a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4044108f70 0x7f404419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 --2- 172.21.15.98:0/1598940872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4044107e20 0x7f404419e120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.557+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 >> 172.21.15.98:0/1598940872 conn(0x7f4044076df0 msgr2=0x7f404410c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.558+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 shutdown_connections 2024-03-16T12:18:43.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.558+0000 7f4048e00640 1 -- 172.21.15.98:0/1598940872 wait complete. 2024-03-16T12:18:43.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 -- 172.21.15.98:0/2648319041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c108df0 msgr2=0x7fe65c109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 --2- 172.21.15.98:0/2648319041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c108df0 0x7fe65c109270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe648009900 tx=0x7fe64802bc40 comp rx=0 tx=0).stop 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 -- 172.21.15.98:0/2648319041 shutdown_connections 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 --2- 172.21.15.98:0/2648319041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c1097b0 0x7fe65c110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 --2- 172.21.15.98:0/2648319041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c108df0 0x7fe65c109270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 --2- 172.21.15.98:0/2648319041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c107c00 0x7fe65c108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 -- 172.21.15.98:0/2648319041 >> 172.21.15.98:0/2648319041 conn(0x7fe65c076ba0 msgr2=0x7fe65c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 -- 172.21.15.98:0/2648319041 shutdown_connections 2024-03-16T12:18:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.774+0000 7fe663000640 1 -- 172.21.15.98:0/2648319041 wait complete. 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 Processor -- start 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 -- start start 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c107c00 0x7fe65c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 0x7fe65c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c1138b0 con 0x7fe65c107c00 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c113730 con 0x7fe65c1097b0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe663000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c113a30 con 0x7fe65c108df0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe662600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe661c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c107c00 0x7fe65c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe662600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33406/0 (socket says 172.21.15.98:33406) 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.775+0000 7fe662600640 1 -- 172.21.15.98:0/1138691730 learned_addr learned my addr 172.21.15.98:0/1138691730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe661200640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 0x7fe65c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 -- 172.21.15.98:0/1138691730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 msgr2=0x7fe65c19e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 0x7fe65c19e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 -- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c107c00 msgr2=0x7fe65c19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c107c00 0x7fe65c19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 -- 172.21.15.98:0/1138691730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65c1a5b30 con 0x7fe65c1097b0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe661200640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 0x7fe65c19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe662600640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe65400d950 tx=0x7fe65400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654017030 con 0x7fe65c1097b0 2024-03-16T12:18:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe654004020 con 0x7fe65c1097b0 2024-03-16T12:18:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654003c00 con 0x7fe65c1097b0 2024-03-16T12:18:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.776+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65c1a5e20 con 0x7fe65c1097b0 2024-03-16T12:18:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.777+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65c1a6290 con 0x7fe65c1097b0 2024-03-16T12:18:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.778+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65c10eec0 con 0x7fe65c1097b0 2024-03-16T12:18:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.780+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe65401a070 con 0x7fe65c1097b0 2024-03-16T12:18:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.781+0000 7fe65a000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 0x7fe6440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.781+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fe6540ce4e0 con 0x7fe65c1097b0 2024-03-16T12:18:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.781+0000 7fe661c00640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 0x7fe6440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.782+0000 7fe661c00640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 0x7fe6440a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe65c108c50 tx=0x7fe650009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.785+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe654021070 con 0x7fe65c1097b0 2024-03-16T12:18:43.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:43.981+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fe65c0624f0 con 0x7fe65c1097b0 2024-03-16T12:18:44.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: pgmap v316: 97 pgs: 12 creating+peering, 36 unknown, 49 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 801 B/s wr, 1 op/s 2024-03-16T12:18:44.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T12:18:44.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T12:18:44.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T12:18:44.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1138691730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:44 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: pgmap v316: 97 pgs: 12 creating+peering, 36 unknown, 49 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 801 B/s wr, 1 op/s 2024-03-16T12:18:44.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T12:18:44.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T12:18:44.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T12:18:44.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1138691730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:44 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: pgmap v316: 97 pgs: 12 creating+peering, 36 unknown, 49 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 801 B/s wr, 1 op/s 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1138691730' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T12:18:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.551+0000 7fe65a000640 1 -- 172.21.15.98:0/1138691730 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fe65409b360 con 0x7fe65c1097b0 2024-03-16T12:18:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 4 allow_ec_overwrites to true 2024-03-16T12:18:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 msgr2=0x7fe6440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 0x7fe6440a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe65c108c50 tx=0x7fe650009670 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 msgr2=0x7fe65c1a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe65400d950 tx=0x7fe65400de10 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 shutdown_connections 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe6440a2880 0x7fe6440a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe65c1097b0 0x7fe65c1a54c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe65c108df0 0x7fe65c19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 --2- 172.21.15.98:0/1138691730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe65c107c00 0x7fe65c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.557+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 >> 172.21.15.98:0/1138691730 conn(0x7fe65c076ba0 msgr2=0x7fe65c10c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.558+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 shutdown_connections 2024-03-16T12:18:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.558+0000 7fe663000640 1 -- 172.21.15.98:0/1138691730 wait complete. 2024-03-16T12:18:44.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-16T12:18:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 -- 172.21.15.98:0/3731394374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4102260 msgr2=0x7f84e4104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 --2- 172.21.15.98:0/3731394374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4102260 0x7f84e4104650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009860 tx=0x7f84d402bb70 comp rx=0 tx=0).stop 2024-03-16T12:18:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 -- 172.21.15.98:0/3731394374 shutdown_connections 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 --2- 172.21.15.98:0/3731394374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e41074c0 0x7f84e4109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 --2- 172.21.15.98:0/3731394374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4104b90 0x7f84e4106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 --2- 172.21.15.98:0/3731394374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4102260 0x7f84e4104650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 -- 172.21.15.98:0/3731394374 >> 172.21.15.98:0/3731394374 conn(0x7f84e40fbc50 msgr2=0x7f84e40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 -- 172.21.15.98:0/3731394374 shutdown_connections 2024-03-16T12:18:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.773+0000 7f84e9e00640 1 -- 172.21.15.98:0/3731394374 wait complete. 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.774+0000 7f84e9e00640 1 Processor -- start 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.774+0000 7f84e9e00640 1 -- start start 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4102260 0x7f84e4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e41074c0 0x7f84e41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84e410d490 con 0x7f84e41074c0 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84e410d310 con 0x7f84e4104b90 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e9e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84e410d610 con 0x7f84e4102260 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e41074c0 0x7f84e41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33428/0 (socket says 172.21.15.98:33428) 2024-03-16T12:18:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 -- 172.21.15.98:0/4157932831 learned_addr learned my addr 172.21.15.98:0/4157932831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84e2a00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4102260 0x7f84e4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 -- 172.21.15.98:0/4157932831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4102260 msgr2=0x7f84e4199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4102260 0x7f84e4199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 -- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e41074c0 msgr2=0x7f84e41a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e41074c0 0x7f84e41a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.775+0000 7f84dbe00640 1 -- 172.21.15.98:0/4157932831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e41a19b0 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84dbe00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00e9f0 tx=0x7f84cc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc00ccc0 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84cc005500 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc010690 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e41a1ca0 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.776+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e41a21a0 con 0x7f84e4104b90 2024-03-16T12:18:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.777+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e4109010 con 0x7f84e4104b90 2024-03-16T12:18:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.779+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f84cc0046c0 con 0x7f84e4104b90 2024-03-16T12:18:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.780+0000 7f84e8a00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 0x7f84c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.780+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f84cc0cf070 con 0x7f84e4104b90 2024-03-16T12:18:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.780+0000 7f84e2a00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 0x7f84c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.781+0000 7f84e2a00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 0x7f84c80a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84e419b270 tx=0x7f84d4034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:44.784+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84cc09be90 con 0x7f84e4104b90 2024-03-16T12:18:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.021+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f84e4062410 con 0x7f84e4104b90 2024-03-16T12:18:45.254 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:18:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: 2024-03-16T12:18:44.919+0000 7fb9af21a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T12:18:45.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.570+0000 7f84e8a00640 1 -- 172.21.15.98:0/4157932831 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f84cc09b790 con 0x7f84e4104b90 2024-03-16T12:18:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:added data pool 4 to fsmap 2024-03-16T12:18:45.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 msgr2=0x7f84c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:45.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 0x7f84c80a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84e419b270 tx=0x7f84d4034ab0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 msgr2=0x7f84e419a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00e9f0 tx=0x7f84cc00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 shutdown_connections 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84c80a2880 0x7f84c80a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e41074c0 0x7f84e41a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e4104b90 0x7f84e419a2e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 --2- 172.21.15.98:0/4157932831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e4102260 0x7f84e4199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.574+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 >> 172.21.15.98:0/4157932831 conn(0x7f84e40fbc50 msgr2=0x7f84e41019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.575+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 shutdown_connections 2024-03-16T12:18:45.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.575+0000 7f84e9e00640 1 -- 172.21.15.98:0/4157932831 wait complete. 2024-03-16T12:18:45.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 -- 172.21.15.98:0/3175876376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 msgr2=0x7f4334109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 --2- 172.21.15.98:0/3175876376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f4334109470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4320009900 tx=0x7f432002bc40 comp rx=0 tx=0).stop 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 -- 172.21.15.98:0/3175876376 shutdown_connections 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 --2- 172.21.15.98:0/3175876376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43341099b0 0x7f4334110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 --2- 172.21.15.98:0/3175876376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f4334109470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 --2- 172.21.15.98:0/3175876376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4334107e00 0x7f4334108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 -- 172.21.15.98:0/3175876376 >> 172.21.15.98:0/3175876376 conn(0x7f4334076dc0 msgr2=0x7f43340791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 -- 172.21.15.98:0/3175876376 shutdown_connections 2024-03-16T12:18:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.800+0000 7f433bc00640 1 -- 172.21.15.98:0/3175876376 wait complete. 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.801+0000 7f433bc00640 1 Processor -- start 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 -- start start 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4334107e00 0x7f433419e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43341099b0 0x7f43341a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4334113ad0 con 0x7f4334107e00 2024-03-16T12:18:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4334113950 con 0x7f4334108ff0 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f433bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4334113c50 con 0x7f43341099b0 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f4339200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4334107e00 0x7f433419e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f4333e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f4333e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33452/0 (socket says 172.21.15.98:33452) 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f4333e00640 1 -- 172.21.15.98:0/107024654 learned_addr learned my addr 172.21.15.98:0/107024654 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.802+0000 7f4339c00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43341099b0 0x7f43341a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 -- 172.21.15.98:0/107024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43341099b0 msgr2=0x7f43341a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43341099b0 0x7f43341a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 -- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4334107e00 msgr2=0x7f433419e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4334107e00 0x7f433419e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 -- 172.21.15.98:0/107024654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43341a5d30 con 0x7f4334108ff0 2024-03-16T12:18:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4333e00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f432002c8f0 tx=0x7f432002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320041070 con 0x7f4334108ff0 2024-03-16T12:18:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4320004450 con 0x7f4334108ff0 2024-03-16T12:18:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43341a5fc0 con 0x7f4334108ff0 2024-03-16T12:18:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.803+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320034440 con 0x7f4334108ff0 2024-03-16T12:18:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.804+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43341a6430 con 0x7f4334108ff0 2024-03-16T12:18:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.804+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433410f0c0 con 0x7f4334108ff0 2024-03-16T12:18:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.806+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4320044070 con 0x7f4334108ff0 2024-03-16T12:18:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.808+0000 7f4331600640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 0x7f431c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.808+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f43200f0770 con 0x7f4334108ff0 2024-03-16T12:18:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.808+0000 7f4339200640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 0x7f431c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.809+0000 7f4339200640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 0x7f431c0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4334108e50 tx=0x7f4328005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:45.812+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43200bcf20 con 0x7f4334108ff0 2024-03-16T12:18:45.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:45 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T12:18:45.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:45 smithi173 ceph-mon[36632]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T12:18:45.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:45 smithi173 ceph-mon[36632]: pgmap v319: 129 pgs: 23 creating+peering, 14 unknown, 92 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-16T12:18:45.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:45 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4157932831' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:45.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:45 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:45.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:45 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T12:18:45.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:45 smithi098 ceph-mon[31633]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T12:18:45.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:45 smithi098 ceph-mon[31633]: pgmap v319: 129 pgs: 23 creating+peering, 14 unknown, 92 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-16T12:18:45.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:45 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4157932831' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:45.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:45 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:46.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:45 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T12:18:46.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:45 smithi153 ceph-mon[36901]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T12:18:46.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:45 smithi153 ceph-mon[36901]: pgmap v319: 129 pgs: 23 creating+peering, 14 unknown, 92 active+clean; 578 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-03-16T12:18:46.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:45 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4157932831' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:46.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:45 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T12:18:46.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.048+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4334062480 con 0x7f4334108ff0 2024-03-16T12:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:18:45.024618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24599,"qdb_cluster":[24599]},"id":1}]} 2024-03-16T12:18:46.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.049+0000 7f4331600640 1 -- 172.21.15.98:0/107024654 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f43200c0fc0 con 0x7f4334108ff0 2024-03-16T12:18:46.033 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 msgr2=0x7f431c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 0x7f431c0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4334108e50 tx=0x7f4328005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 msgr2=0x7f433419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f432002c8f0 tx=0x7f432002c660 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 shutdown_connections 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f431c0a2880 0x7f431c0a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43341099b0 0x7f43341a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4334108ff0 0x7f433419e650 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 --2- 172.21.15.98:0/107024654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4334107e00 0x7f433419e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 >> 172.21.15.98:0/107024654 conn(0x7f4334076dc0 msgr2=0x7f433410c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:46.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 shutdown_connections 2024-03-16T12:18:46.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.052+0000 7f433bc00640 1 -- 172.21.15.98:0/107024654 wait complete. 2024-03-16T12:18:46.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 -- 172.21.15.98:0/2576867097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf28103a00 msgr2=0x7fbf28103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 --2- 172.21.15.98:0/2576867097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf28103a00 0x7fbf28103e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009900 tx=0x7fbf1c02bc50 comp rx=0 tx=0).stop 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 -- 172.21.15.98:0/2576867097 shutdown_connections 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 --2- 172.21.15.98:0/2576867097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf281055b0 0x7fbf2810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 --2- 172.21.15.98:0/2576867097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28104bf0 0x7fbf28105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 --2- 172.21.15.98:0/2576867097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf28103a00 0x7fbf28103e20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 -- 172.21.15.98:0/2576867097 >> 172.21.15.98:0/2576867097 conn(0x7fbf280fef90 msgr2=0x7fbf281013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:46.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 -- 172.21.15.98:0/2576867097 shutdown_connections 2024-03-16T12:18:46.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.281+0000 7fbf30200640 1 -- 172.21.15.98:0/2576867097 wait complete. 2024-03-16T12:18:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.282+0000 7fbf30200640 1 Processor -- start 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.282+0000 7fbf30200640 1 -- start start 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.282+0000 7fbf30200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 0x7fbf28199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf30200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28104bf0 0x7fbf2819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf30200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf30200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810f6c0 con 0x7fbf28104bf0 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf30200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810f540 con 0x7fbf281055b0 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf30200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2810f840 con 0x7fbf28103a00 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28104bf0 0x7fbf2819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 0x7fbf28199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33472/0 (socket says 172.21.15.98:33472) 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 -- 172.21.15.98:0/3899076494 learned_addr learned my addr 172.21.15.98:0/3899076494 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:46.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 0x7fbf28199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38890/0 (socket says 172.21.15.98:38890) 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 -- 172.21.15.98:0/3899076494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 msgr2=0x7fbf28199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 0x7fbf28199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 -- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28104bf0 msgr2=0x7fbf2819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28104bf0 0x7fbf2819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 -- 172.21.15.98:0/3899076494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf281a1930 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.283+0000 7fbf2e200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000eb10 tx=0x7fbf2000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.284+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20004b20 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.284+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf20004cc0 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.284+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf281a1c20 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.284+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20010760 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.284+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf281a2140 con 0x7fbf281055b0 2024-03-16T12:18:46.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.285+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2810ad20 con 0x7fbf281055b0 2024-03-16T12:18:46.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.287+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbf20005020 con 0x7fbf281055b0 2024-03-16T12:18:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.288+0000 7fbf26000640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 0x7fbf100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.288+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbf200cf270 con 0x7fbf281055b0 2024-03-16T12:18:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.288+0000 7fbf2d800640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 0x7fbf100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.289+0000 7fbf2d800640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 0x7fbf100a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c02c190 tx=0x7fbf1c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:46.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.292+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf2009c070 con 0x7fbf281055b0 2024-03-16T12:18:46.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.495+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf28062480 con 0x7fbf281055b0 2024-03-16T12:18:46.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.497+0000 7fbf26000640 1 -- 172.21.15.98:0/3899076494 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fbf2009b970 con 0x7fbf281055b0 2024-03-16T12:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 msgr2=0x7fbf100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 0x7fbf100a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c02c190 tx=0x7fbf1c000bc0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 msgr2=0x7fbf281a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000eb10 tx=0x7fbf2000c2d0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 shutdown_connections 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbf100a2880 0x7fbf100a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf281055b0 0x7fbf281a1290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28104bf0 0x7fbf2819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 --2- 172.21.15.98:0/3899076494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28103a00 0x7fbf28199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 >> 172.21.15.98:0/3899076494 conn(0x7fbf280fef90 msgr2=0x7fbf28107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.499+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 shutdown_connections 2024-03-16T12:18:46.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.500+0000 7fbf30200640 1 -- 172.21.15.98:0/3899076494 wait complete. 2024-03-16T12:18:46.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:18:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 -- 172.21.15.98:0/1837299638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b01027d0 msgr2=0x7f96b0104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1837299638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b01027d0 0x7f96b0104c40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f969c009900 tx=0x7f969c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 -- 172.21.15.98:0/1837299638 shutdown_connections 2024-03-16T12:18:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1837299638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 0x7f96b0105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1837299638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b01027d0 0x7f96b0104c40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1837299638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00ffe60 0x7f96b0102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 -- 172.21.15.98:0/1837299638 >> 172.21.15.98:0/1837299638 conn(0x7f96b00fba60 msgr2=0x7f96b00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 -- 172.21.15.98:0/1837299638 shutdown_connections 2024-03-16T12:18:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.721+0000 7f96b4e00640 1 -- 172.21.15.98:0/1837299638 wait complete. 2024-03-16T12:18:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 Processor -- start 2024-03-16T12:18:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 -- start start 2024-03-16T12:18:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 0x7f96b019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 0x7f96b01a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01138b0 con 0x7f96b00ffe60 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0113730 con 0x7f96b01027d0 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.722+0000 7f96b4e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0113a30 con 0x7f96b0105180 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96af400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 0x7f96b019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33490/0 (socket says 172.21.15.98:33490) 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 -- 172.21.15.98:0/2971590582 learned_addr learned my addr 172.21.15.98:0/2971590582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96af400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 0x7f96b019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52658/0 (socket says 172.21.15.98:52658) 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96afe00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 0x7f96b01a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 -- 172.21.15.98:0/2971590582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 msgr2=0x7f96b01a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 0x7f96b01a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 -- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 msgr2=0x7f96b019dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 0x7f96b019dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 -- 172.21.15.98:0/2971590582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b01a5ac0 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.723+0000 7f96aea00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.724+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c043020 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.724+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f969c0054e0 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.724+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b01a5d50 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.724+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c003c40 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.724+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01a6260 con 0x7f96b01027d0 2024-03-16T12:18:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.725+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b00fd550 con 0x7f96b01027d0 2024-03-16T12:18:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.726+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f969c046020 con 0x7f96b01027d0 2024-03-16T12:18:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.728+0000 7f96a7e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.728+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f969c0f0c60 con 0x7f96b01027d0 2024-03-16T12:18:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.728+0000 7f96af400640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.729+0000 7f96af400640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009bd0 tx=0x7f96a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:46.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.731+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f969c04d020 con 0x7f96b01027d0 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/107024654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:46.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:46 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3899076494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:46.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T12:18:46.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T12:18:46.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T12:18:46.862 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:46.862 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/107024654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:46.862 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:46 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3899076494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:46.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.935+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96b0062410 con 0x7f96b01027d0 2024-03-16T12:18:46.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.937+0000 7f96a7e00640 1 -- 172.21.15.98:0/2971590582 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7f969c0bdac0 con 0x7f96b01027d0 2024-03-16T12:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 msgr2=0x7f96940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009bd0 tx=0x7f96a0005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 msgr2=0x7f96b019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c02c650 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 shutdown_connections 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0105180 0x7f96b01a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b01027d0 0x7f96b019e3e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 --2- 172.21.15.98:0/2971590582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b00ffe60 0x7f96b019dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 >> 172.21.15.98:0/2971590582 conn(0x7f96b00fba60 msgr2=0x7f96b01007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 shutdown_connections 2024-03-16T12:18:46.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:46.941+0000 7f96b4e00640 1 -- 172.21.15.98:0/2971590582 wait complete. 2024-03-16T12:18:46.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/107024654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:46 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3899076494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 -- 172.21.15.98:0/3372654063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41031a0 msgr2=0x7f5ad4109a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/3372654063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41031a0 0x7f5ad4109a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009ca0 tx=0x7f5acc02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 -- 172.21.15.98:0/3372654063 shutdown_connections 2024-03-16T12:18:47.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/3372654063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41031a0 0x7f5ad4109a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/3372654063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 0x7f5ad4102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/3372654063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41015f0 0x7f5ad4101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 -- 172.21.15.98:0/3372654063 >> 172.21.15.98:0/3372654063 conn(0x7f5ad40fcb40 msgr2=0x7f5ad40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:47.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 -- 172.21.15.98:0/3372654063 shutdown_connections 2024-03-16T12:18:47.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.163+0000 7f5ad3e00640 1 -- 172.21.15.98:0/3372654063 wait complete. 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.164+0000 7f5ad3e00640 1 Processor -- start 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 -- start start 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 0x7f5ad4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 0x7f5ad4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad410d270 con 0x7f5ad41031a0 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad410d0f0 con 0x7f5ad41015f0 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad410d3f0 con 0x7f5ad41027e0 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45134/0 (socket says 172.21.15.98:45134) 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad2a00640 1 -- 172.21.15.98:0/1318220705 learned_addr learned my addr 172.21.15.98:0/1318220705 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad3400640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 0x7f5ad4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad2000640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 0x7f5ad4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.165+0000 7f5ad2a00640 1 -- 172.21.15.98:0/1318220705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 msgr2=0x7f5ad4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad2a00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 0x7f5ad4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad2a00640 1 -- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 msgr2=0x7f5ad4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad2a00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 0x7f5ad4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad2a00640 1 -- 172.21.15.98:0/1318220705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad40749a0 con 0x7f5ad41015f0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad3400640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 0x7f5ad4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad2a00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400e9c0 tx=0x7f5ac400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400cd80 con 0x7f5ad41015f0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac40035f0 con 0x7f5ad41015f0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4010690 con 0x7f5ad41015f0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad41a1970 con 0x7f5ad41015f0 2024-03-16T12:18:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.166+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a1f10 con 0x7f5ad41015f0 2024-03-16T12:18:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.167+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad41088b0 con 0x7f5ad41015f0 2024-03-16T12:18:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.169+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5ac40037a0 con 0x7f5ad41015f0 2024-03-16T12:18:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.170+0000 7f5acb400640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 0x7f5ab80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.170+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5ac40cf090 con 0x7f5ad41015f0 2024-03-16T12:18:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.171+0000 7f5ad2000640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 0x7f5ab80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.172+0000 7f5ad2000640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 0x7f5ab80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4075300 tx=0x7f5ac000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.174+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ac409be90 con 0x7f5ad41015f0 2024-03-16T12:18:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.405+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad402d1b0 con 0x7f5ad41015f0 2024-03-16T12:18:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.406+0000 7f5acb400640 1 -- 172.21.15.98:0/1318220705 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f5ac409b790 con 0x7f5ad41015f0 2024-03-16T12:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:18:45.024618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24599,"qdb_cluster":[24599]},"id":1}]} 2024-03-16T12:18:47.389 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-03-16T12:18:47.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 msgr2=0x7f5ab80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 0x7f5ab80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4075300 tx=0x7f5ac000b390 comp rx=0 tx=0).stop 2024-03-16T12:18:47.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 msgr2=0x7f5ad4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400e9c0 tx=0x7f5ac400ee80 comp rx=0 tx=0).stop 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 shutdown_connections 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5ab80a2880 0x7f5ab80a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ad41031a0 0x7f5ad4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ad41027e0 0x7f5ad4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 --2- 172.21.15.98:0/1318220705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad41015f0 0x7f5ad4075590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.408+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 >> 172.21.15.98:0/1318220705 conn(0x7f5ad40fcb40 msgr2=0x7f5ad4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.409+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 shutdown_connections 2024-03-16T12:18:47.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.409+0000 7f5ad3e00640 1 -- 172.21.15.98:0/1318220705 wait complete. 2024-03-16T12:18:47.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:18:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 -- 172.21.15.98:0/2280645265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8478101600 msgr2=0x7f8478101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 --2- 172.21.15.98:0/2280645265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8478101600 0x7f8478101a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f846c009900 tx=0x7f846c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 -- 172.21.15.98:0/2280645265 shutdown_connections 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 --2- 172.21.15.98:0/2280645265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f8478109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 --2- 172.21.15.98:0/2280645265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84781027f0 0x7f8478102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 --2- 172.21.15.98:0/2280645265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8478101600 0x7f8478101a20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 -- 172.21.15.98:0/2280645265 >> 172.21.15.98:0/2280645265 conn(0x7f84780fcb70 msgr2=0x7f84780fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 -- 172.21.15.98:0/2280645265 shutdown_connections 2024-03-16T12:18:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.616+0000 7f847ee00640 1 -- 172.21.15.98:0/2280645265 wait complete. 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 Processor -- start 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 -- start start 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8478101600 0x7f8478197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84781027f0 0x7f8478197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f847810d2d0 con 0x7f84781027f0 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f847810d150 con 0x7f84781031b0 2024-03-16T12:18:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.618+0000 7f847ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f847810d450 con 0x7f8478101600 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84781027f0 0x7f8478197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45160/0 (socket says 172.21.15.98:45160) 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 -- 172.21.15.98:0/4211738123 learned_addr learned my addr 172.21.15.98:0/4211738123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847da00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8478101600 0x7f8478197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 -- 172.21.15.98:0/4211738123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8478101600 msgr2=0x7f8478197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8478101600 0x7f8478197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 -- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84781027f0 msgr2=0x7f8478197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84781027f0 0x7f8478197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 -- 172.21.15.98:0/4211738123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847819f530 con 0x7f84781031b0 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.619+0000 7f847e400640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f847000efd0 tx=0x7f847000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.620+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8470003b70 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.620+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8470003d10 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.620+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847819f820 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.620+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84700109f0 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.620+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847819fd70 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.621+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8478108920 con 0x7f84781031b0 2024-03-16T12:18:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.623+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8470004020 con 0x7f84781031b0 2024-03-16T12:18:47.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.625+0000 7f8476000640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 0x7f84600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:47.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.625+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84700cf0f0 con 0x7f84781031b0 2024-03-16T12:18:47.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.625+0000 7f847da00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 0x7f84600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:47.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.625+0000 7f847da00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 0x7f84600a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f846c02c170 tx=0x7f846c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:47.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.628+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f847009bef0 con 0x7f84781031b0 2024-03-16T12:18:47.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:47 smithi098 ceph-mon[31633]: pgmap v321: 129 pgs: 17 creating+peering, 2 unknown, 110 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 9 op/s 2024-03-16T12:18:47.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:47 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2971590582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.840 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:47 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1318220705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:47 smithi173 ceph-mon[36632]: pgmap v321: 129 pgs: 17 creating+peering, 2 unknown, 110 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 9 op/s 2024-03-16T12:18:47.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:47 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2971590582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.841 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:47 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1318220705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:47.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.823+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84780623e0 con 0x7f84781031b0 2024-03-16T12:18:47.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.825+0000 7f8476000640 1 -- 172.21.15.98:0/4211738123 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7f847009b7f0 con 0x7f84781031b0 2024-03-16T12:18:47.842 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:47.842 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:18:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.828+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 msgr2=0x7f84600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.828+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 0x7f84600a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f846c02c170 tx=0x7f846c02df90 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 msgr2=0x7f847819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f847000efd0 tx=0x7f847000c6a0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 shutdown_connections 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84600a2880 0x7f84600a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84781031b0 0x7f847819eec0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84781027f0 0x7f8478197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 --2- 172.21.15.98:0/4211738123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8478101600 0x7f8478197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 >> 172.21.15.98:0/4211738123 conn(0x7f84780fcb70 msgr2=0x7f8478105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 shutdown_connections 2024-03-16T12:18:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:47.829+0000 7f847ee00640 1 -- 172.21.15.98:0/4211738123 wait complete. 2024-03-16T12:18:47.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T12:18:40.454968+0000', 'modified': '2024-03-16T12:18:45.024618+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/542416413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 542416413}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 542416413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24599, 'qdb_cluster': [24599]} 2024-03-16T12:18:47.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-16T12:18:47.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T12:18:48.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:47 smithi153 ceph-mon[36901]: pgmap v321: 129 pgs: 17 creating+peering, 2 unknown, 110 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 9 op/s 2024-03-16T12:18:48.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:47 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2971590582' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:48.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:47 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1318220705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:18:48.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 -- 172.21.15.98:0/1081056156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 msgr2=0x7fbcd8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 --2- 172.21.15.98:0/1081056156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8102e60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009900 tx=0x7fbcc402bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 -- 172.21.15.98:0/1081056156 shutdown_connections 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 --2- 172.21.15.98:0/1081056156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 --2- 172.21.15.98:0/1081056156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8102e60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 --2- 172.21.15.98:0/1081056156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 0x7fbcd8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 -- 172.21.15.98:0/1081056156 >> 172.21.15.98:0/1081056156 conn(0x7fbcd80fcd80 msgr2=0x7fbcd80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 -- 172.21.15.98:0/1081056156 shutdown_connections 2024-03-16T12:18:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.062+0000 7fbce0200640 1 -- 172.21.15.98:0/1081056156 wait complete. 2024-03-16T12:18:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 Processor -- start 2024-03-16T12:18:48.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 -- start start 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 0x7fbcd8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd810d4c0 con 0x7fbcd81033a0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd810d340 con 0x7fbcd81029e0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.063+0000 7fbce0200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd810d640 con 0x7fbcd81017f0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45176/0 (socket says 172.21.15.98:45176) 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 -- 172.21.15.98:0/1093607941 learned_addr learned my addr 172.21.15.98:0/1093607941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 0x7fbcd8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcde200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 -- 172.21.15.98:0/1093607941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 msgr2=0x7fbcd8075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 0x7fbcd8075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 -- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 msgr2=0x7fbcd8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 -- 172.21.15.98:0/1093607941 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80749a0 con 0x7fbcd81029e0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcde200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.064+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4000c00 tx=0x7fbcc402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.065+0000 7fbcd6000640 1 -- 172.21.15.98:0/1093607941 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4043020 con 0x7fbcd81029e0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.065+0000 7fbcd6000640 1 -- 172.21.15.98:0/1093607941 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcc40054e0 con 0x7fbcd81029e0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.065+0000 7fbcd6000640 1 -- 172.21.15.98:0/1093607941 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4003c40 con 0x7fbcd81029e0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.065+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd81a1c80 con 0x7fbcd81029e0 2024-03-16T12:18:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.065+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd81a21d0 con 0x7fbcd81029e0 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbcd6000640 1 -- 172.21.15.98:0/1093607941 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbcc4046020 con 0x7fbcd81029e0 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 Processor -- start 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 -- start start 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f22a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 0x56235f22d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56235f189c50 con 0x56235f225b50 2024-03-16T12:18:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56235f1329d0 con 0x56235f22acc0 2024-03-16T12:18:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.068+0000 7fbce3008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56235f185cf0 con 0x56235f225160 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcdce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 0x56235f22d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f22a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55480/0 (socket says 172.21.15.98:55480) 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 -- 172.21.15.98:0/1723590975 learned_addr learned my addr 172.21.15.98:0/1723590975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 -- 172.21.15.98:0/1723590975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 msgr2=0x56235f22d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 0x56235f22d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 -- 172.21.15.98:0/1723590975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 msgr2=0x56235f22a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f22a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 -- 172.21.15.98:0/1723590975 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56235f14cf10 con 0x56235f225160 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 0x56235f22d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcd6000640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 0x7fbcc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.069+0000 7fbcde200640 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009fd0 tx=0x7fbcd00108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97a23453d5de019 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbcd6000640 1 -- 172.21.15.98:0/1093607941 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbcc40f0df0 con 0x7fbcd81029e0 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 0x7fbcc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbccaa00640 1 -- 172.21.15.98:0/1723590975 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0011760 con 0x56235f225160 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbccaa00640 1 -- 172.21.15.98:0/1723590975 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcd0011de0 con 0x56235f225160 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbccaa00640 1 -- 172.21.15.98:0/1723590975 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd000e3c0 con 0x56235f225160 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 0x7fbcc00a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x56235f225f50 tx=0x7fbccc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbce3008740 1 -- 172.21.15.98:0/1723590975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 msgr2=0x56235f225580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.070+0000 7fbce3008740 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009fd0 tx=0x7fbcd00108e0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 -- 172.21.15.98:0/1723590975 shutdown_connections 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f22acc0 0x56235f22d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f22a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 --2- 172.21.15.98:0/1723590975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225160 0x56235f225580 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 -- 172.21.15.98:0/1723590975 >> 172.21.15.98:0/1723590975 conn(0x56235f2206d0 msgr2=0x56235f220ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 -- 172.21.15.98:0/1723590975 shutdown_connections 2024-03-16T12:18:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 -- 172.21.15.98:0/1723590975 wait complete. 2024-03-16T12:18:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 -- start start 2024-03-16T12:18:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.071+0000 7fbce3008740 1 Processor -- start 2024-03-16T12:18:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 0x56235f22f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f236840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56235f22dc70 con 0x56235f225160 2024-03-16T12:18:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56235f145e20 con 0x56235f225b50 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56235f22ddf0 con 0x56235f22acc0 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbce3008740 10 client.0 _ll_register_callbacks cb 0x56235f1457f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54990/0 (socket says 172.21.15.98:54990) 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 -- 172.21.15.98:0/519092200 learned_addr learned my addr 172.21.15.98:0/519092200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcde200640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f236840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdce00640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 0x56235f22f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 -- 172.21.15.98:0/519092200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 msgr2=0x56235f236840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f236840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 -- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 msgr2=0x56235f22f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 0x56235f22f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.072+0000 7fbcdd800640 1 -- 172.21.15.98:0/519092200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56235ee9e2d0 con 0x56235f225160 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbcdce00640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 0x56235f22f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbcdd800640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbccc004620 tx=0x7fbccc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0146fdb3b5a193 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbcbf400640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc01f630 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbccc01f7d0 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.073+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcd0011de0 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbccc01f9d0 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 10 client.14595 fetch_fsmap learned FSMap version 17 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56235ef686b0 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcd0011de0 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fbccc027070 con 0x56235f225160 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 10 client.14595 fetch_fsmap finished waiting for FSMap version 17 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 10 client.14595 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 20 client.14595 populate_metadata read hostname 'smithi098' 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f23c160 0x56235f23d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 4 client.14595 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56235f1e72b0 con 0x56235f23c160 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.074+0000 7fbcdce00640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f23c160 0x56235f23d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.075+0000 7fbcdce00640 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f23c160 0x56235f23d5d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4224c00d6ce77533 server_cookie=4c1fa387b1d4f059 in_seq=0 out_seq=0 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.075+0000 7fbcbf400640 10 client.14595 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.075+0000 7fbcbf400640 1 -- 172.21.15.98:0/519092200 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56235f1e72b0 con 0x56235f23c160 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbcbf400640 10 client.14595 handle_command_reply: tid=0 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 client.14595 shutdown 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 msgr2=0x56235f22f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbccc004620 tx=0x7fbccc004730 comp rx=0 tx=0).stop 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 shutdown_connections 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f23c160 0x56235f23d5d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f236840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225b50 0x56235f22f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbcbf400640 0 client.14595 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 --2- 172.21.15.98:0/519092200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225160 0x56235f22f2c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 >> 172.21.15.98:0/519092200 conn(0x56235f2206d0 msgr2=0x56235f21fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 shutdown_connections 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 1 -- 172.21.15.98:0/519092200 wait complete. 2024-03-16T12:18:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.076+0000 7fbce3008740 20 client.14595 trim_cache size 0 max 16384 2024-03-16T12:18:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 Processor -- start 2024-03-16T12:18:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 -- start start 2024-03-16T12:18:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 0x56235f236800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f22acc0 0x56235f22f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56235f23b910 con 0x56235f22acc0 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56235f23bfe0 con 0x56235f225160 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.095+0000 7fbce3008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56235f23c6b0 con 0x56235f225b50 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcde200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f22acc0 0x56235f22f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45216/0 (socket says 172.21.15.98:45216) 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 -- 172.21.15.98:0/1205382772 learned_addr learned my addr 172.21.15.98:0/1205382772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 0x56235f236800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 -- 172.21.15.98:0/1205382772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 msgr2=0x56235f236800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 0x56235f236800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 -- 172.21.15.98:0/1205382772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f22acc0 msgr2=0x56235f22f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f22acc0 0x56235f22f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 -- 172.21.15.98:0/1205382772 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56235f1f6750 con 0x56235f225160 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdce00640 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 0x56235f236800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcdd800640 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009710 tx=0x7fbccc003fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d9850efdf77e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcca000640 1 -- 172.21.15.98:0/1205382772 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc01f630 con 0x56235f225160 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcca000640 1 -- 172.21.15.98:0/1205382772 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbccc01f7d0 con 0x56235f225160 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.096+0000 7fbcca000640 1 -- 172.21.15.98:0/1205382772 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc029800 con 0x56235f225160 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- 172.21.15.98:0/1205382772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 msgr2=0x56235f22eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009710 tx=0x7fbccc003fe0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- 172.21.15.98:0/1205382772 shutdown_connections 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f22acc0 0x56235f22f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f225b50 0x56235f236800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 --2- 172.21.15.98:0/1205382772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f22eed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- 172.21.15.98:0/1205382772 >> 172.21.15.98:0/1205382772 conn(0x56235f2206d0 msgr2=0x56235f220ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- 172.21.15.98:0/1205382772 shutdown_connections 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- 172.21.15.98:0/1205382772 wait complete. 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.097+0000 7fbce3008740 1 -- start start 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 Processor -- start 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f232fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f235f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56235f23d450 con 0x56235f225b50 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56235f23cd80 con 0x56235f225160 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56235f23db20 con 0x56235f22acc0 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbce3008740 10 client.0 _ll_register_callbacks cb 0x7fbccc0044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:18:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbcdd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45224/0 (socket says 172.21.15.98:45224) 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbcdd800640 1 -- 172.21.15.98:0/3438538220 learned_addr learned my addr 172.21.15.98:0/3438538220 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.098+0000 7fbcdce00640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f232fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcde200640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f235f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 -- 172.21.15.98:0/3438538220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 msgr2=0x56235f235f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f235f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 -- 172.21.15.98:0/3438538220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 msgr2=0x56235f232fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f232fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 -- 172.21.15.98:0/3438538220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56235f236570 con 0x56235f225160 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcde200640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f235f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcdd800640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbccc012e20 tx=0x7fbccc0297f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a3bad5d1ef858af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcbf400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc023040 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbccc01f7d0 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbccc01d6f0 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbccc01f630 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbce3008740 10 client.24713 fetch_fsmap learned FSMap version 17 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.099+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56235f2a29b0 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbccc01f7d0 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fbccc037070 con 0x56235f225160 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 10 client.24713 fetch_fsmap finished waiting for FSMap version 17 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 10 client.24713 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 20 client.24713 populate_metadata read hostname 'smithi098' 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f2a7310 0x56235f2a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 4 client.24713 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "status"}) -- 0x56235f197190 con 0x56235f2a7310 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.100+0000 7fbcdce00640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f2a7310 0x56235f2a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbcbf400640 10 client.24713 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbcdce00640 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f2a7310 0x56235f2a6020 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8921d269f43900 server_cookie=f9817834dff3b063 in_seq=0 out_seq=0 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbcbf400640 1 -- 172.21.15.98:0/3438538220 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56235f197190 con 0x56235f2a7310 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbcbf400640 10 client.24713 handle_command_reply: tid=0 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 client.24713 shutdown 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 msgr2=0x56235f232a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbccc012e20 tx=0x7fbccc0297f0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 shutdown_connections 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56235f2a7310 0x56235f2a6020 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56235f22acc0 0x56235f235f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbcbf400640 0 client.24713 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56235f225b50 0x56235f232fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 --2- 172.21.15.98:0/3438538220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56235f225160 0x56235f232a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 >> 172.21.15.98:0/3438538220 conn(0x56235f2206d0 msgr2=0x56235f22eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 shutdown_connections 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 1 -- 172.21.15.98:0/3438538220 wait complete. 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.101+0000 7fbce3008740 20 client.24713 trim_cache size 0 max 16384 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:18:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24599, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.0765713720000001, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 17, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 89, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 13.749589719999999 2024-03-16T12:18:48.092 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 msgr2=0x7fbcc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 0x7fbcc00a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x56235f225f50 tx=0x7fbccc005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 msgr2=0x7fbcd8073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4000c00 tx=0x7fbcc402c650 comp rx=0 tx=0).stop 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 shutdown_connections 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbcc00a2800 0x7fbcc00a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcd81033a0 0x7fbcd8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd81029e0 0x7fbcd8073bd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 --2- 172.21.15.98:0/1093607941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd81017f0 0x7fbcd8075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 >> 172.21.15.98:0/1093607941 conn(0x7fbcd80fcd80 msgr2=0x7fbcd8105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 shutdown_connections 2024-03-16T12:18:48.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:48.102+0000 7fbce0200640 1 -- 172.21.15.98:0/1093607941 wait complete. 2024-03-16T12:18:48.102 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T12:18:48.102 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:48.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T12:18:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:18:48.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:48 smithi173 ceph-mon[36632]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T12:18:48.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:48 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:18:48.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:48 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4211738123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:48.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:48 smithi098 ceph-mon[31633]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T12:18:48.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:48 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:18:48.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:48 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4211738123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:49.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:48 smithi153 ceph-mon[36901]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T12:18:49.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:48 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:18:49.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:48 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4211738123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:49.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:49 smithi098 ceph-mon[31633]: pgmap v322: 129 pgs: 14 creating+peering, 115 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-03-16T12:18:50.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:49 smithi153 ceph-mon[36901]: pgmap v322: 129 pgs: 14 creating+peering, 115 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-03-16T12:18:50.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:49 smithi173 ceph-mon[36632]: pgmap v322: 129 pgs: 14 creating+peering, 115 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-03-16T12:18:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 -- 172.21.15.98:0/567117389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b806a920 msgr2=0x7f13b810f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 --2- 172.21.15.98:0/567117389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b806a920 0x7f13b810f160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13b000b0a0 tx=0x7f13b000b3b0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 -- 172.21.15.98:0/567117389 shutdown_connections 2024-03-16T12:18:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 --2- 172.21.15.98:0/567117389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b806a920 0x7f13b810f160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 --2- 172.21.15.98:0/567117389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b8108320 0x7f13b8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 --2- 172.21.15.98:0/567117389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 0x7f13b806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 -- 172.21.15.98:0/567117389 >> 172.21.15.98:0/567117389 conn(0x7f13b8078100 msgr2=0x7f13b8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 -- 172.21.15.98:0/567117389 shutdown_connections 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.898+0000 7f13bf400700 1 -- 172.21.15.98:0/567117389 wait complete. 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.899+0000 7f13bf400700 1 Processor -- start 2024-03-16T12:18:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 -- start start 2024-03-16T12:18:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 0x7f13b819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b806a920 0x7f13b819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81125c0 con 0x7f13b806a920 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13b8112440 con 0x7f13b8108320 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bf400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13b8112740 con 0x7f13b8069f80 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bd400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13b7e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b806a920 0x7f13b819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bd400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:45248/0 (socket says 172.21.15.98:45248) 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.900+0000 7f13bd400700 1 -- 172.21.15.98:0/2370115781 learned_addr learned my addr 172.21.15.98:0/2370115781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bca00700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 0x7f13b819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 -- 172.21.15.98:0/2370115781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 msgr2=0x7f13b819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 0x7f13b819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 -- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b806a920 msgr2=0x7f13b819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b806a920 0x7f13b819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:50.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 -- 172.21.15.98:0/2370115781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b8199080 con 0x7f13b8108320 2024-03-16T12:18:50.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bd400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13b0006070 tx=0x7f13b0004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:50.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0021070 con 0x7f13b8108320 2024-03-16T12:18:50.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13b0003c00 con 0x7f13b8108320 2024-03-16T12:18:50.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.901+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b8199310 con 0x7f13b8108320 2024-03-16T12:18:50.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.902+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b00047c0 con 0x7f13b8108320 2024-03-16T12:18:50.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.902+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b81a55e0 con 0x7f13b8108320 2024-03-16T12:18:50.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.903+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b8068f90 con 0x7f13b8108320 2024-03-16T12:18:50.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.904+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f13b0007980 con 0x7f13b8108320 2024-03-16T12:18:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.905+0000 7f13b5600700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 0x7f13a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.906+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f13b000f020 con 0x7f13b8108320 2024-03-16T12:18:50.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.906+0000 7f13bca00700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 0x7f13a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:50.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.906+0000 7f13bca00700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 0x7f13a00a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13b80ff950 tx=0x7f13ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:50.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:50.910+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13b002a020 con 0x7f13b8108320 2024-03-16T12:18:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.182+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f13b8199e00 con 0x7f13b8108320 2024-03-16T12:18:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.183+0000 7f13b5600700 1 -- 172.21.15.98:0/2370115781 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f13b009cf50 con 0x7f13b8108320 2024-03-16T12:18:51.166 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-16T12:18:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.185+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 msgr2=0x7f13a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.185+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 0x7f13a00a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13b80ff950 tx=0x7f13ac009260 comp rx=0 tx=0).stop 2024-03-16T12:18:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 msgr2=0x7f13b8198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13b0006070 tx=0x7f13b0004cf0 comp rx=0 tx=0).stop 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 shutdown_connections 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f13a00a2910 0x7f13a00a4dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b8108320 0x7f13b8198920 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b8069f80 0x7f13b819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 --2- 172.21.15.98:0/2370115781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b806a920 0x7f13b819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 >> 172.21.15.98:0/2370115781 conn(0x7f13b8078100 msgr2=0x7f13b80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 shutdown_connections 2024-03-16T12:18:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:51.186+0000 7f13bf400700 1 -- 172.21.15.98:0/2370115781 wait complete. 2024-03-16T12:18:51.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:51 smithi098 ceph-mon[31633]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-16T12:18:51.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:51 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2370115781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:18:52.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:51 smithi173 ceph-mon[36632]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-16T12:18:52.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:51 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2370115781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T12:18:52.153 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T12:18:52.154 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T12:18:52.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:52.154 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:18:52.209 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42096sec preferred_lft 42096sec 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-03-16T12:18:52.210 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:18:52.212 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T12:18:52.212 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:18:52.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:51 smithi153 ceph-mon[36901]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-16T12:18:52.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:51 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2370115781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:18:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:18:52.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:52.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:52.743 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T12:18:52.820 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-16T12:18:52.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:52.823 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-03-16T12:18:52.878 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-03-16T12:18:52.878 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T12:18:52.878 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T12:18:52.878 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T12:18:52.878 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T12:18:52.878 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:18:52.878 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:18:52.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T12:18:52.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T12:18:52.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T12:18:52.879 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:18:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:18:53.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:53.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:53.704 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-16T12:18:53.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:53.759 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-03-16T12:18:53.810 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:18:53.811 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:18:53.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:53.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-16T12:18:53.811 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:18:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:18:53.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:53 smithi098 ceph-mon[31633]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-16T12:18:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:54.017 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-16T12:18:54.017 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-16T12:18:54.018 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:18:54.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:53 smithi173 ceph-mon[36632]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-16T12:18:54.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:18:54.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:53 smithi153 ceph-mon[36901]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-16T12:18:54.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:54.359 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:18:54.359 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:18:54.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-03-16T12:18:54.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T12:18:54.360 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:18:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:18:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:54.581 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-16T12:18:54.582 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T12:18:54.582 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:54.623 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T12:18:54.624 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T12:18:54.624 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:54.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-16T12:18:54.803 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T12:18:54.856 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-03-16T12:18:54.856 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-03-16T12:18:54.856 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-03-16T12:18:54.856 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-03-16T12:18:54.856 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-03-16T12:18:54.857 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/1de6dd084341ae09402ac108ef31c8894d2f6a27b316ccd7947fb6e29fc6b17a/merged 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/cd1201717d99b49614b0ab4c6ff0d602a9c63a6caba92bab64d2a42542dd69aa/merged 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/5e39474d4091204c702c8a48b0c88d17ba22862db03148ba95f8c3bf0768ecf8/merged 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/351a1bffdf5463228fa7297813357e4e24fa189591e27b53ea87f6b5aeac1736/merged 2024-03-16T12:18:54.858 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3fdcd4848093ec27b18a2e2444d2594b921a73f720b9bf6649b299e8f12887a9/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/23ba5fb53ed0b4fbd315da8b238e2949be5eefad04641e090eae44dc56764330/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/44de55244f189f83a7a4c2b0e1650e4e5103e33006f2f10a9a9bc2dfa97ff7d5/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/849d0b4287b5ce839b784cc617e9f95cecafbf19ab903d6ca8b695e916156819/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/df36a0ce355b793307696f174c14f47d9f3c17b52ead629a271ebcb7c0d1f666/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/a7ba7be87cbf090895649102087388e66cd280145dcf7254381e23457f821af8/merged 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:54.859 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:54.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:54.860 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-16T12:18:54.909 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-16T12:18:54.909 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-16T12:18:54.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-16T12:18:54.990 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T12:18:55.035 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-03-16T12:18:55.036 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/1de6dd084341ae09402ac108ef31c8894d2f6a27b316ccd7947fb6e29fc6b17a/merged 2024-03-16T12:18:55.037 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/cd1201717d99b49614b0ab4c6ff0d602a9c63a6caba92bab64d2a42542dd69aa/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/5e39474d4091204c702c8a48b0c88d17ba22862db03148ba95f8c3bf0768ecf8/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/351a1bffdf5463228fa7297813357e4e24fa189591e27b53ea87f6b5aeac1736/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3fdcd4848093ec27b18a2e2444d2594b921a73f720b9bf6649b299e8f12887a9/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/23ba5fb53ed0b4fbd315da8b238e2949be5eefad04641e090eae44dc56764330/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/44de55244f189f83a7a4c2b0e1650e4e5103e33006f2f10a9a9bc2dfa97ff7d5/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/849d0b4287b5ce839b784cc617e9f95cecafbf19ab903d6ca8b695e916156819/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/df36a0ce355b793307696f174c14f47d9f3c17b52ead629a271ebcb7c0d1f666/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/a7ba7be87cbf090895649102087388e66cd280145dcf7254381e23457f821af8/merged 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:55.038 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:55.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:55.039 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-16T12:18:55.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:2024-03-16T12:18:55.139+0000 7fb67c377480 -1 init, newargv = 0x557e55d63a60 newargc=15 2024-03-16T12:18:55.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[75793]: starting ceph client 2024-03-16T12:18:55.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[75793]: starting fuse 2024-03-16T12:18:56.071 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:55 smithi173 ceph-mon[36632]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 2 op/s 2024-03-16T12:18:56.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-03-16T12:18:56.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:55 smithi098 ceph-mon[31633]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 2 op/s 2024-03-16T12:18:56.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:55 smithi153 ceph-mon[36901]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 2 op/s 2024-03-16T12:18:56.326 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T12:18:56.426 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-03-16T12:18:56.426 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-03-16T12:18:56.426 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-03-16T12:18:56.427 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/1de6dd084341ae09402ac108ef31c8894d2f6a27b316ccd7947fb6e29fc6b17a/merged 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/cd1201717d99b49614b0ab4c6ff0d602a9c63a6caba92bab64d2a42542dd69aa/merged 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/5e39474d4091204c702c8a48b0c88d17ba22862db03148ba95f8c3bf0768ecf8/merged 2024-03-16T12:18:56.428 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/351a1bffdf5463228fa7297813357e4e24fa189591e27b53ea87f6b5aeac1736/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3fdcd4848093ec27b18a2e2444d2594b921a73f720b9bf6649b299e8f12887a9/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/23ba5fb53ed0b4fbd315da8b238e2949be5eefad04641e090eae44dc56764330/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/44de55244f189f83a7a4c2b0e1650e4e5103e33006f2f10a9a9bc2dfa97ff7d5/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/849d0b4287b5ce839b784cc617e9f95cecafbf19ab903d6ca8b695e916156819/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/df36a0ce355b793307696f174c14f47d9f3c17b52ead629a271ebcb7c0d1f666/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/a7ba7be87cbf090895649102087388e66cd280145dcf7254381e23457f821af8/merged 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:18:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:56.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:56.430 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-03-16T12:18:56.482 INFO:teuthology.orchestra.run.smithi098.stdout:118 2024-03-16T12:18:56.482 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-16T12:18:56.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-03-16T12:18:56.482 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-03-16T12:18:56.482 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-03-16T12:18:56.482 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-03-16T12:18:56.483 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-03-16T12:18:56.484 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-03-16T12:18:56.484 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-16T12:18:56.484 DEBUG:teuthology.orchestra.run.smithi098:> 2024-03-16T12:18:56.484 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.admin")) 2024-03-16T12:18:56.484 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-03-16T12:18:56.654 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.admin.75793.asok 2024-03-16T12:18:56.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:56.672 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75793.asok 2024-03-16T12:18:56.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:56.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75793.asok status 2024-03-16T12:18:56.904 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:18:56.904 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "admin", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "75793", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14613, 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 14613 2024-03-16T12:18:56.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3781792289 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3781792289 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.14613 192.168.0.1:0/3781792289", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/3781792289", 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-03-16T12:18:56.906 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 17, 2024-03-16T12:18:56.907 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 89, 2024-03-16T12:18:56.907 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-03-16T12:18:56.907 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-03-16T12:18:56.907 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-03-16T12:18:56.907 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:18:56.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:56.920 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-03-16T12:18:56.920 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:56.970 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-03-16T12:18:56.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:56.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:18:56.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:18:57.064 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-16T12:18:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T12:18:57.201 INFO:teuthology.orchestra.run.smithi098.stdout:# file: . 2024-03-16T12:18:57.201 INFO:teuthology.orchestra.run.smithi098.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-16T12:18:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:57.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:18:57.212 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:18:57.212 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:18:57.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.541+0000 7fbebd800640 1 -- 172.21.15.98:0/558296324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 msgr2=0x7fbeb8109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.541+0000 7fbebd800640 1 --2- 172.21.15.98:0/558296324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb8109290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009900 tx=0x7fbea402bc50 comp rx=0 tx=0).stop 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 -- 172.21.15.98:0/558296324 shutdown_connections 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 --2- 172.21.15.98:0/558296324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 0x7fbeb8110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 --2- 172.21.15.98:0/558296324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb8109290 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 --2- 172.21.15.98:0/558296324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb8108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 -- 172.21.15.98:0/558296324 >> 172.21.15.98:0/558296324 conn(0x7fbeb8076bb0 msgr2=0x7fbeb8078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 -- 172.21.15.98:0/558296324 shutdown_connections 2024-03-16T12:18:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.542+0000 7fbebd800640 1 -- 172.21.15.98:0/558296324 wait complete. 2024-03-16T12:18:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 Processor -- start 2024-03-16T12:18:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 -- start start 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 0x7fbeb81a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8113880 con 0x7fbeb81097d0 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8113700 con 0x7fbeb8108e10 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8113a00 con 0x7fbeb8107c20 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbeb7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbebce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 0x7fbeb81a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbeb7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:49968/0 (socket says 172.21.15.98:49968) 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbeb7400640 1 -- 172.21.15.98:0/2047525753 learned_addr learned my addr 172.21.15.98:0/2047525753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.543+0000 7fbeb7e00640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 -- 172.21.15.98:0/2047525753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 msgr2=0x7fbeb819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 -- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 msgr2=0x7fbeb81a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 0x7fbeb81a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 -- 172.21.15.98:0/2047525753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb81a5b80 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7e00640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb7400640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009230 tx=0x7fbea4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4042020 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea40044d0 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea403e460 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.544+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb81a5e10 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.545+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb81a6280 con 0x7fbeb8108e10 2024-03-16T12:18:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.546+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb810ef40 con 0x7fbeb8108e10 2024-03-16T12:18:57.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.548+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbea4045020 con 0x7fbeb8108e10 2024-03-16T12:18:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.549+0000 7fbeb4c00640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 0x7fbe9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.550+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbea40f0df0 con 0x7fbeb8108e10 2024-03-16T12:18:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.550+0000 7fbeb7e00640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 0x7fbe9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.550+0000 7fbeb7e00640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 0x7fbe9c0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbea8005d00 tx=0x7fbea8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:57.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.554+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbea404c020 con 0x7fbeb8108e10 2024-03-16T12:18:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.756+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbeb80623e0 con 0x7fbeb8108e10 2024-03-16T12:18:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.758+0000 7fbeb4c00640 1 -- 172.21.15.98:0/2047525753 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fbea40bdbf0 con 0x7fbeb8108e10 2024-03-16T12:18:57.741 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:18:57.741 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:18:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 msgr2=0x7fbe9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 0x7fbe9c0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbea8005d00 tx=0x7fbea8005c90 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 msgr2=0x7fbeb819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009230 tx=0x7fbea4005d50 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 shutdown_connections 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbe9c0a2800 0x7fbe9c0a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeb81097d0 0x7fbeb81a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb8108e10 0x7fbeb819e4a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 --2- 172.21.15.98:0/2047525753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb8107c20 0x7fbeb819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 >> 172.21.15.98:0/2047525753 conn(0x7fbeb8076bb0 msgr2=0x7fbeb810c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 shutdown_connections 2024-03-16T12:18:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.761+0000 7fbebd800640 1 -- 172.21.15.98:0/2047525753 wait complete. 2024-03-16T12:18:57.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:18:57.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:18:57.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.984+0000 7f4961000640 1 -- 172.21.15.98:0/4582433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10d3e0 msgr2=0x7f495c10d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.984+0000 7f4961000640 1 --2- 172.21.15.98:0/4582433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10d3e0 0x7f495c10d860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4948009900 tx=0x7f494802bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 -- 172.21.15.98:0/4582433 shutdown_connections 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 --2- 172.21.15.98:0/4582433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10dda0 0x7f495c114630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 --2- 172.21.15.98:0/4582433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10d3e0 0x7f495c10d860 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 --2- 172.21.15.98:0/4582433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c10c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 -- 172.21.15.98:0/4582433 >> 172.21.15.98:0/4582433 conn(0x7f495c072b30 msgr2=0x7f495c072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 -- 172.21.15.98:0/4582433 shutdown_connections 2024-03-16T12:18:57.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.985+0000 7f4961000640 1 -- 172.21.15.98:0/4582433 wait complete. 2024-03-16T12:18:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 Processor -- start 2024-03-16T12:18:57.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 -- start start 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c1a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 0x7f495c1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f495c117ed0 con 0x7f495c10d3e0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f495c117d50 con 0x7f495c10dda0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f4961000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f495c118050 con 0x7f495c10c1f0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f495a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f495a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:49976/0 (socket says 172.21.15.98:49976) 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.986+0000 7f495a800640 1 -- 172.21.15.98:0/4263561520 learned_addr learned my addr 172.21.15.98:0/4263561520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4959e00640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c1a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4959400640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 0x7f495c1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 -- 172.21.15.98:0/4263561520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 msgr2=0x7f495c1a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c1a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 -- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 msgr2=0x7f495c1a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 0x7f495c1a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 -- 172.21.15.98:0/4263561520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c1aa160 con 0x7f495c10dda0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4959400640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 0x7f495c1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4959e00640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c1a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f495a800640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f495400eaa0 tx=0x7f495400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495400cdb0 con 0x7f495c10dda0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4954005020 con 0x7f495c10dda0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540116b0 con 0x7f495c10dda0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495c1aa450 con 0x7f495c10dda0 2024-03-16T12:18:57.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.987+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495c1aa9a0 con 0x7f495c10dda0 2024-03-16T12:18:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.989+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495c113510 con 0x7f495c10dda0 2024-03-16T12:18:57.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.990+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4954005560 con 0x7f495c10dda0 2024-03-16T12:18:57.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.991+0000 7f4952a00640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 0x7f49400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:57.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.992+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f49540cee90 con 0x7f495c10dda0 2024-03-16T12:18:57.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.992+0000 7f4959e00640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 0x7f49400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:57.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.992+0000 7f4959e00640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 0x7f49400a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f494c005d00 tx=0x7f494c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:57.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:57.995+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4954016030 con 0x7f495c10dda0 2024-03-16T12:18:58.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:57 smithi098 ceph-mon[31633]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s wr, 0 op/s 2024-03-16T12:18:58.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:57 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2047525753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.195+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f495c0623e0 con 0x7f495c10dda0 2024-03-16T12:18:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.196+0000 7f4952a00640 1 -- 172.21.15.98:0/4263561520 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f495409bcf0 con 0x7f495c10dda0 2024-03-16T12:18:58.179 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:18:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 msgr2=0x7f49400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 0x7f49400a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f494c005d00 tx=0x7f494c005c90 comp rx=0 tx=0).stop 2024-03-16T12:18:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 msgr2=0x7f495c1a9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f495400eaa0 tx=0x7f495400ef60 comp rx=0 tx=0).stop 2024-03-16T12:18:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 shutdown_connections 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f49400a2880 0x7f49400a4d40 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f494c005d00 tx=0x7f494c005c90 comp rx=0 tx=0).stop 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f495c10dda0 0x7f495c1a9af0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f495c10d3e0 0x7f495c1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 --2- 172.21.15.98:0/4263561520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f495c10c1f0 0x7f495c1a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 >> 172.21.15.98:0/4263561520 conn(0x7f495c072b30 msgr2=0x7f495c1106e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.199+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 shutdown_connections 2024-03-16T12:18:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.200+0000 7f4961000640 1 -- 172.21.15.98:0/4263561520 wait complete. 2024-03-16T12:18:58.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:18:58.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:57 smithi153 ceph-mon[36901]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s wr, 0 op/s 2024-03-16T12:18:58.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:57 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2047525753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:58.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:57 smithi173 ceph-mon[36632]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s wr, 0 op/s 2024-03-16T12:18:58.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:57 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2047525753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:18:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 -- 172.21.15.98:0/591603583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 msgr2=0x7f94c8109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 --2- 172.21.15.98:0/591603583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c8109810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009ca0 tx=0x7f94c002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 -- 172.21.15.98:0/591603583 shutdown_connections 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 --2- 172.21.15.98:0/591603583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c8109810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 --2- 172.21.15.98:0/591603583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 0x7f94c8106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 --2- 172.21.15.98:0/591603583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 -- 172.21.15.98:0/591603583 >> 172.21.15.98:0/591603583 conn(0x7f94c80fbb50 msgr2=0x7f94c80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.423+0000 7f94c7e00640 1 -- 172.21.15.98:0/591603583 shutdown_connections 2024-03-16T12:18:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.424+0000 7f94c7e00640 1 -- 172.21.15.98:0/591603583 wait complete. 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.424+0000 7f94c7e00640 1 Processor -- start 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 -- start start 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 0x7f94c8197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810d380 con 0x7f94c8102160 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810d200 con 0x7f94c81073c0 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810d500 con 0x7f94c8104a90 2024-03-16T12:18:58.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:49998/0 (socket says 172.21.15.98:49998) 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 -- 172.21.15.98:0/54296379 learned_addr learned my addr 172.21.15.98:0/54296379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94be000640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 0x7f94c8197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 -- 172.21.15.98:0/54296379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 msgr2=0x7f94c8197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 0x7f94c8197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.425+0000 7f94c7400640 1 -- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 msgr2=0x7f94c8197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c7400640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c7400640 1 -- 172.21.15.98:0/54296379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c819f5a0 con 0x7f94c81073c0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c6a00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c7400640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94c002c630 tx=0x7f94c0032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0041020 con 0x7f94c81073c0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94c0034cd0 con 0x7f94c81073c0 2024-03-16T12:18:58.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0039040 con 0x7f94c81073c0 2024-03-16T12:18:58.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c819f830 con 0x7f94c81073c0 2024-03-16T12:18:58.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.426+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c819fd20 con 0x7f94c81073c0 2024-03-16T12:18:58.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.427+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c80fd640 con 0x7f94c81073c0 2024-03-16T12:18:58.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.429+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f94c0044020 con 0x7f94c81073c0 2024-03-16T12:18:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.430+0000 7f94bfe00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 0x7f94ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.430+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f94c00f0ac0 con 0x7f94c81073c0 2024-03-16T12:18:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.430+0000 7f94c6a00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 0x7f94ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.431+0000 7f94c6a00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 0x7f94ac0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009fc0 tx=0x7f94b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:58.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.439+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94c004b020 con 0x7f94c81073c0 2024-03-16T12:18:58.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.638+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94c80623e0 con 0x7f94c81073c0 2024-03-16T12:18:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.639+0000 7f94bfe00640 1 -- 172.21.15.98:0/54296379 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f94c00b7e20 con 0x7f94c81073c0 2024-03-16T12:18:58.622 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 msgr2=0x7f94ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 0x7f94ac0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009fc0 tx=0x7f94b8005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 msgr2=0x7f94c819ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94c002c630 tx=0x7f94c0032ea0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 shutdown_connections 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94ac0a2800 0x7f94ac0a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c81073c0 0x7f94c819ef30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c8104a90 0x7f94c8197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 --2- 172.21.15.98:0/54296379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c8102160 0x7f94c8197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 >> 172.21.15.98:0/54296379 conn(0x7f94c80fbb50 msgr2=0x7f94c8102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 shutdown_connections 2024-03-16T12:18:58.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.642+0000 7f94c7e00640 1 -- 172.21.15.98:0/54296379 wait complete. 2024-03-16T12:18:58.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 -- 172.21.15.98:0/1912058260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 msgr2=0x7fcb301009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 --2- 172.21.15.98:0/1912058260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb301009e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009ca0 tx=0x7fcb2002c100 comp rx=0 tx=0).stop 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 -- 172.21.15.98:0/1912058260 shutdown_connections 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 --2- 172.21.15.98:0/1912058260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 0x7fcb30108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 --2- 172.21.15.98:0/1912058260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 --2- 172.21.15.98:0/1912058260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb301009e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 -- 172.21.15.98:0/1912058260 >> 172.21.15.98:0/1912058260 conn(0x7fcb300fbb50 msgr2=0x7fcb300fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 -- 172.21.15.98:0/1912058260 shutdown_connections 2024-03-16T12:18:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.865+0000 7fcb34a00640 1 -- 172.21.15.98:0/1912058260 wait complete. 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.866+0000 7fcb34a00640 1 Processor -- start 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.866+0000 7fcb34a00640 1 -- start start 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 0x7fcb301030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010c230 con 0x7fcb301005c0 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010c0b0 con 0x7fcb301017b0 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb34a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010c3b0 con 0x7fcb30102170 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49730/0 (socket says 172.21.15.98:49730) 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 -- 172.21.15.98:0/2821229503 learned_addr learned my addr 172.21.15.98:0/2821229503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2f200640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 0x7fcb301030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 -- 172.21.15.98:0/2821229503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 msgr2=0x7fcb301030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 0x7fcb301030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 -- 172.21.15.98:0/2821229503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 msgr2=0x7fcb30102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb2e800640 1 -- 172.21.15.98:0/2821229503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb30103990 con 0x7fcb301005c0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.867+0000 7fcb25e00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb2e800640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f80 tx=0x7fcb20035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20039040 con 0x7fcb301005c0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb20044070 con 0x7fcb301005c0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20004430 con 0x7fcb301005c0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb301a1af0 con 0x7fcb301005c0 2024-03-16T12:18:58.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.868+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb301a1fb0 con 0x7fcb301005c0 2024-03-16T12:18:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.869+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb301a2180 con 0x7fcb301005c0 2024-03-16T12:18:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.873+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcb200045d0 con 0x7fcb301005c0 2024-03-16T12:18:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.874+0000 7fcb27e00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 0x7fcb140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:58.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.874+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcb200f1cd0 con 0x7fcb301005c0 2024-03-16T12:18:58.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.874+0000 7fcb25e00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 0x7fcb140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:58.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.875+0000 7fcb25e00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 0x7fcb140a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb30104270 tx=0x7fcb18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:58.878+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb200beb30 con 0x7fcb301005c0 2024-03-16T12:18:58.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:58 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4263561520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:58.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:58 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/54296379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:18:59.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.075+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb301a2470 con 0x7fcb301005c0 2024-03-16T12:18:59.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.076+0000 7fcb27e00640 1 -- 172.21.15.98:0/2821229503 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fcb200be430 con 0x7fcb301005c0 2024-03-16T12:18:59.059 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 msgr2=0x7fcb140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 0x7fcb140a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb30104270 tx=0x7fcb18005f90 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 msgr2=0x7fcb30104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f80 tx=0x7fcb20035d10 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 shutdown_connections 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb140a2800 0x7fcb140a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb30102170 0x7fcb301030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb301017b0 0x7fcb30102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 --2- 172.21.15.98:0/2821229503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb301005c0 0x7fcb30104500 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 >> 172.21.15.98:0/2821229503 conn(0x7fcb300fbb50 msgr2=0x7fcb300fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 shutdown_connections 2024-03-16T12:18:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.079+0000 7fcb34a00640 1 -- 172.21.15.98:0/2821229503 wait complete. 2024-03-16T12:18:59.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:18:59.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:58 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4263561520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:59.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:58 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/54296379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 -- 172.21.15.98:0/3960291484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf0104c10 msgr2=0x7f4cf0105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 --2- 172.21.15.98:0/3960291484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf0104c10 0x7f4cf0105090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009900 tx=0x7f4cdc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 -- 172.21.15.98:0/3960291484 shutdown_connections 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 --2- 172.21.15.98:0/3960291484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf01055d0 0x7f4cf010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 --2- 172.21.15.98:0/3960291484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf0104c10 0x7f4cf0105090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 --2- 172.21.15.98:0/3960291484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 -- 172.21.15.98:0/3960291484 >> 172.21.15.98:0/3960291484 conn(0x7f4cf00fef90 msgr2=0x7f4cf01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 -- 172.21.15.98:0/3960291484 shutdown_connections 2024-03-16T12:18:59.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.314+0000 7f4cf5000640 1 -- 172.21.15.98:0/3960291484 wait complete. 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.315+0000 7f4cf5000640 1 Processor -- start 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.315+0000 7f4cf5000640 1 -- start start 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0104c10 0x7f4cf019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf01055d0 0x7f4cf01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf010f6a0 con 0x7f4cf01055d0 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf010f520 con 0x7f4cf0103a20 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cf5000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf010f820 con 0x7f4cf0104c10 2024-03-16T12:18:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50050/0 (socket says 172.21.15.98:50050) 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 -- 172.21.15.98:0/955172690 learned_addr learned my addr 172.21.15.98:0/955172690 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cee800640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf01055d0 0x7f4cf01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4ced400640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0104c10 0x7f4cf019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 -- 172.21.15.98:0/955172690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0104c10 msgr2=0x7f4cf019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0104c10 0x7f4cf019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 -- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf01055d0 msgr2=0x7f4cf01a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf01055d0 0x7f4cf01a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.316+0000 7f4cede00640 1 -- 172.21.15.98:0/955172690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf01a1960 con 0x7f4cf0103a20 2024-03-16T12:18:59.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4cede00640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e9c0 tx=0x7f4ce000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000cde0 con 0x7f4cf0103a20 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce0010910 con 0x7f4cf0103a20 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0018a00 con 0x7f4cf0103a20 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf01a1c50 con 0x7f4cf0103a20 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.317+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf01a2170 con 0x7f4cf0103a20 2024-03-16T12:18:59.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.318+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf010ad40 con 0x7f4cf0103a20 2024-03-16T12:18:59.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.320+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4ce001f020 con 0x7f4cf0103a20 2024-03-16T12:18:59.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.322+0000 7f4ce6a00640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 0x7f4cd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.322+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4ce00cef30 con 0x7f4cf0103a20 2024-03-16T12:18:59.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.322+0000 7f4ced400640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 0x7f4cd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.323+0000 7f4ced400640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 0x7f4cd40a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc006010 tx=0x7f4cdc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:59.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.326+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ce009bd90 con 0x7f4cf0103a20 2024-03-16T12:18:59.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:58 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4263561520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:18:59.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:58 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/54296379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:18:59.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.525+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4cf0062480 con 0x7f4cf0103a20 2024-03-16T12:18:59.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.525+0000 7f4ce6a00640 1 -- 172.21.15.98:0/955172690 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f4ce009b690 con 0x7f4cf0103a20 2024-03-16T12:18:59.508 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:18:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 msgr2=0x7f4cd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 0x7f4cd40a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc006010 tx=0x7f4cdc000bc0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 msgr2=0x7f4cf0199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e9c0 tx=0x7f4ce000ee80 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 shutdown_connections 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4cd40a2800 0x7f4cd40a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4cf01055d0 0x7f4cf01a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0104c10 0x7f4cf019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.528+0000 7f4cf5000640 1 --2- 172.21.15.98:0/955172690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cf0103a20 0x7f4cf0199d40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.529+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 >> 172.21.15.98:0/955172690 conn(0x7f4cf00fef90 msgr2=0x7f4cf0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.529+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 shutdown_connections 2024-03-16T12:18:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.529+0000 7f4cf5000640 1 -- 172.21.15.98:0/955172690 wait complete. 2024-03-16T12:18:59.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T12:18:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.751+0000 7fe79ea00640 1 -- 172.21.15.98:0/3134558979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe798104bf0 msgr2=0x7fe798105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.751+0000 7fe79ea00640 1 --2- 172.21.15.98:0/3134558979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe798104bf0 0x7fe798105070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe780009900 tx=0x7fe78002bc30 comp rx=0 tx=0).stop 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 -- 172.21.15.98:0/3134558979 shutdown_connections 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 --2- 172.21.15.98:0/3134558979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7981055b0 0x7fe79810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 --2- 172.21.15.98:0/3134558979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe798104bf0 0x7fe798105070 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 --2- 172.21.15.98:0/3134558979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 -- 172.21.15.98:0/3134558979 >> 172.21.15.98:0/3134558979 conn(0x7fe7980fef90 msgr2=0x7fe7981013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 -- 172.21.15.98:0/3134558979 shutdown_connections 2024-03-16T12:18:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.752+0000 7fe79ea00640 1 -- 172.21.15.98:0/3134558979 wait complete. 2024-03-16T12:18:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 Processor -- start 2024-03-16T12:18:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 -- start start 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe798104bf0 0x7fe79819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 0x7fe7981a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810f6d0 con 0x7fe798103a00 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810f550 con 0x7fe7981055b0 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.753+0000 7fe79ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810f850 con 0x7fe798104bf0 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49774/0 (socket says 172.21.15.98:49774) 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 -- 172.21.15.98:0/1622249409 learned_addr learned my addr 172.21.15.98:0/1622249409 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe79ca00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 0x7fe7981a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797400640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe798104bf0 0x7fe79819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 -- 172.21.15.98:0/1622249409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe798104bf0 msgr2=0x7fe79819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe798104bf0 0x7fe79819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 -- 172.21.15.98:0/1622249409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 msgr2=0x7fe7981a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 0x7fe7981a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 -- 172.21.15.98:0/1622249409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7981a1950 con 0x7fe798103a00 2024-03-16T12:18:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe79ca00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 0x7fe7981a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.754+0000 7fe797e00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe78800ebc0 tx=0x7fe78800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.755+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788003b70 con 0x7fe798103a00 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.755+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe788003d10 con 0x7fe798103a00 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.755+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7981a1c40 con 0x7fe798103a00 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.755+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788019980 con 0x7fe798103a00 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.755+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7981a2190 con 0x7fe798103a00 2024-03-16T12:18:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.756+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79810ad20 con 0x7fe798103a00 2024-03-16T12:18:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.758+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe788019b20 con 0x7fe798103a00 2024-03-16T12:18:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.759+0000 7fe794c00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 0x7fe77c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:18:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.759+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe7880cf4e0 con 0x7fe798103a00 2024-03-16T12:18:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.759+0000 7fe797400640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 0x7fe77c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:18:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.760+0000 7fe797400640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 0x7fe77c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe78002c240 tx=0x7fe78002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:18:59.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.763+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe78809c340 con 0x7fe798103a00 2024-03-16T12:18:59.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.977+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe79802d1b0 con 0x7fe798103a00 2024-03-16T12:18:59.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.978+0000 7fe794c00640 1 -- 172.21.15.98:0/1622249409 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+588 (secure 0 0 0) 0x7fe78809bc40 con 0x7fe798103a00 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/352508062 2024-03-17T12:10:42.632912+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/653650228 2024-03-17T12:10:08.286891+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/719012091 2024-03-17T12:10:42.632912+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/196846375 2024-03-17T12:10:42.632912+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3496387635 2024-03-17T12:10:08.286891+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3016831032 2024-03-17T12:10:08.286891+0000 2024-03-16T12:18:59.962 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:18:59.963 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 msgr2=0x7fe77c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 0x7fe77c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe78002c240 tx=0x7fe78002df90 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 msgr2=0x7fe798199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe78800ebc0 tx=0x7fe78800c6a0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 shutdown_connections 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe77c0a2800 0x7fe77c0a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7981055b0 0x7fe7981a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe798104bf0 0x7fe79819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 --2- 172.21.15.98:0/1622249409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe798103a00 0x7fe798199d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 >> 172.21.15.98:0/1622249409 conn(0x7fe7980fef90 msgr2=0x7fe798107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 shutdown_connections 2024-03-16T12:18:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:18:59.981+0000 7fe79ea00640 1 -- 172.21.15.98:0/1622249409 wait complete. 2024-03-16T12:18:59.982 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:18:59.982 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:19:00.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:19:00.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:59 smithi098 ceph-mon[31633]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:00.116 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:59 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2821229503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.116 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:18:59 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/955172690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 -- 172.21.15.98:0/661765173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340110060 msgr2=0x7fc340112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 --2- 172.21.15.98:0/661765173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340110060 0x7fc340112450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc338009900 tx=0x7fc33802bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:00.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 -- 172.21.15.98:0/661765173 shutdown_connections 2024-03-16T12:19:00.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 --2- 172.21.15.98:0/661765173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340110060 0x7fc340112450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 --2- 172.21.15.98:0/661765173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc340109140 0x7fc34010fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 --2- 172.21.15.98:0/661765173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3401087e0 0x7fc340108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 -- 172.21.15.98:0/661765173 >> 172.21.15.98:0/661765173 conn(0x7fc34006e9d0 msgr2=0x7fc340070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:00.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 -- 172.21.15.98:0/661765173 shutdown_connections 2024-03-16T12:19:00.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.267+0000 7fc345a00640 1 -- 172.21.15.98:0/661765173 wait complete. 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.268+0000 7fc345a00640 1 Processor -- start 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 -- start start 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 0x7fc340109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc340110060 0x7fc340109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc340115cb0 con 0x7fc3401087e0 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc340115b30 con 0x7fc340109140 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc345a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc340115e30 con 0x7fc340110060 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc33ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc33ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49794/0 (socket says 172.21.15.98:49794) 2024-03-16T12:19:00.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc33e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 0x7fc340109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc33ea00640 1 -- 172.21.15.98:0/2161467172 learned_addr learned my addr 172.21.15.98:0/2161467172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.269+0000 7fc33f400640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc340110060 0x7fc340109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 -- 172.21.15.98:0/2161467172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc340110060 msgr2=0x7fc340109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc340110060 0x7fc340109ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 -- 172.21.15.98:0/2161467172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 msgr2=0x7fc340109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 0x7fc340109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 -- 172.21.15.98:0/2161467172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34010a880 con 0x7fc3401087e0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33e000640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 0x7fc340109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc33ea00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc33000d770 tx=0x7fc33000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330017070 con 0x7fc3401087e0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330009c30 con 0x7fc3401087e0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3401aa350 con 0x7fc3401087e0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330003dd0 con 0x7fc3401087e0 2024-03-16T12:19:00.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.270+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3401aa8a0 con 0x7fc3401087e0 2024-03-16T12:19:00.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:59 smithi153 ceph-mon[36901]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:00.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:59 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2821229503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:18:59 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/955172690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.272+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc34010e930 con 0x7fc3401087e0 2024-03-16T12:19:00.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.273+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc330003f90 con 0x7fc3401087e0 2024-03-16T12:19:00.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.275+0000 7fc337400640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 0x7fc3200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.276+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc3300ce570 con 0x7fc3401087e0 2024-03-16T12:19:00.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.276+0000 7fc33e000640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 0x7fc3200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.276+0000 7fc33e000640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 0x7fc3200a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc34010b1e0 tx=0x7fc328009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:00.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.280+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc330009de0 con 0x7fc3401087e0 2024-03-16T12:19:00.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:59 smithi173 ceph-mon[36632]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:00.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:59 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2821229503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:18:59 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/955172690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.483+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3400623e0 con 0x7fc3401087e0 2024-03-16T12:19:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.487+0000 7fc337400640 1 -- 172.21.15.98:0/2161467172 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fc33009ad80 con 0x7fc3401087e0 2024-03-16T12:19:00.471 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:00.471 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 msgr2=0x7fc3200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 0x7fc3200a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc34010b1e0 tx=0x7fc328009500 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 msgr2=0x7fc34010b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc33000d770 tx=0x7fc33000dc30 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 shutdown_connections 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc3200a2800 0x7fc3200a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc340110060 0x7fc340109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc340109140 0x7fc340109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 --2- 172.21.15.98:0/2161467172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3401087e0 0x7fc34010b470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 >> 172.21.15.98:0/2161467172 conn(0x7fc34006e9d0 msgr2=0x7fc34010bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.490+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 shutdown_connections 2024-03-16T12:19:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.491+0000 7fc345a00640 1 -- 172.21.15.98:0/2161467172 wait complete. 2024-03-16T12:19:00.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:19:00.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:19:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 -- 172.21.15.98:0/542461689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841033a0 msgr2=0x7f8184109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 --2- 172.21.15.98:0/542461689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841033a0 0x7f8184109c30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f817c009ca0 tx=0x7f817c02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 -- 172.21.15.98:0/542461689 shutdown_connections 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 --2- 172.21.15.98:0/542461689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841033a0 0x7f8184109c30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 --2- 172.21.15.98:0/542461689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841029e0 0x7f8184102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 --2- 172.21.15.98:0/542461689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 0x7f8184101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 -- 172.21.15.98:0/542461689 >> 172.21.15.98:0/542461689 conn(0x7f81840fcd80 msgr2=0x7f81840ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 -- 172.21.15.98:0/542461689 shutdown_connections 2024-03-16T12:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.725+0000 7f818be00640 1 -- 172.21.15.98:0/542461689 wait complete. 2024-03-16T12:19:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 Processor -- start 2024-03-16T12:19:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 -- start start 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 0x7f8184197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841033a0 0x7f818419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f818410d450 con 0x7f81841017f0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f818410d2d0 con 0x7f81841029e0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.726+0000 7f818be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f818410d5d0 con 0x7f81841033a0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8189400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 0x7f8184197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50116/0 (socket says 172.21.15.98:50116) 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 -- 172.21.15.98:0/3686869521 learned_addr learned my addr 172.21.15.98:0/3686869521 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8189e00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841033a0 0x7f818419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 -- 172.21.15.98:0/3686869521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841033a0 msgr2=0x7f818419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841033a0 0x7f818419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 -- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 msgr2=0x7f8184197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 0x7f8184197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 -- 172.21.15.98:0/3686869521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818419f6b0 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8188a00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f817000ea60 tx=0x7f817000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.727+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817000cde0 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.728+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81700035f0 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.728+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818419f9a0 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.728+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170010670 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.728+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818419ff40 con 0x7f81841029e0 2024-03-16T12:19:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.729+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8184108b10 con 0x7f81841029e0 2024-03-16T12:19:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.731+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f81700037a0 con 0x7f81841029e0 2024-03-16T12:19:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.732+0000 7f8182000640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 0x7f81680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.733+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f81700cef10 con 0x7f81841029e0 2024-03-16T12:19:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.733+0000 7f8189400640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 0x7f81680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.733+0000 7f8189400640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 0x7f81680a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8184102840 tx=0x7f8174005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:00.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.737+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f817009bd70 con 0x7f81841029e0 2024-03-16T12:19:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.938+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f81840623e0 con 0x7f81841029e0 2024-03-16T12:19:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.938+0000 7f8182000640 1 -- 172.21.15.98:0/3686869521 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f817009b670 con 0x7f81841029e0 2024-03-16T12:19:00.921 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:19:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 msgr2=0x7f81680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 0x7f81680a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8184102840 tx=0x7f8174005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 msgr2=0x7f8184197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f817000ea60 tx=0x7f817000ef20 comp rx=0 tx=0).stop 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 shutdown_connections 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81680a2880 0x7f81680a4d40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81841033a0 0x7f818419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81841029e0 0x7f8184197fd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 --2- 172.21.15.98:0/3686869521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81841017f0 0x7f8184197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 >> 172.21.15.98:0/3686869521 conn(0x7f81840fcd80 msgr2=0x7f8184105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.941+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 shutdown_connections 2024-03-16T12:19:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:00.942+0000 7f818be00640 1 -- 172.21.15.98:0/3686869521 wait complete. 2024-03-16T12:19:00.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:19:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 -- 172.21.15.98:0/1463077465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1029e0 msgr2=0x7f227c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 --2- 172.21.15.98:0/1463077465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1029e0 0x7f227c102e60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2264009900 tx=0x7f226402bc40 comp rx=0 tx=0).stop 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 -- 172.21.15.98:0/1463077465 shutdown_connections 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 --2- 172.21.15.98:0/1463077465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1033a0 0x7f227c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 --2- 172.21.15.98:0/1463077465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1029e0 0x7f227c102e60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 --2- 172.21.15.98:0/1463077465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1017f0 0x7f227c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 -- 172.21.15.98:0/1463077465 >> 172.21.15.98:0/1463077465 conn(0x7f227c0fcd60 msgr2=0x7f227c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 -- 172.21.15.98:0/1463077465 shutdown_connections 2024-03-16T12:19:01.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.178+0000 7f2281600640 1 -- 172.21.15.98:0/1463077465 wait complete. 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 Processor -- start 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 -- start start 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1017f0 0x7f227c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1033a0 0x7f227c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f227c10d4a0 con 0x7f227c1017f0 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f227c10d320 con 0x7f227c1029e0 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f2281600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f227c10d620 con 0x7f227c1033a0 2024-03-16T12:19:01.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f227aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1017f0 0x7f227c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.179+0000 7f227a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50122/0 (socket says 172.21.15.98:50122) 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 -- 172.21.15.98:0/1774695115 learned_addr learned my addr 172.21.15.98:0/1774695115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227b400640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1033a0 0x7f227c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 -- 172.21.15.98:0/1774695115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1033a0 msgr2=0x7f227c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1033a0 0x7f227c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 -- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1017f0 msgr2=0x7f227c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1017f0 0x7f227c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 -- 172.21.15.98:0/1774695115 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c0749a0 con 0x7f227c1029e0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f227a000640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f226402c8f0 tx=0x7f226402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264041070 con 0x7f227c1029e0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2264004450 con 0x7f227c1029e0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264034440 con 0x7f227c1029e0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.180+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c1a1c00 con 0x7f227c1029e0 2024-03-16T12:19:01.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.181+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c1a2100 con 0x7f227c1029e0 2024-03-16T12:19:01.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.182+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c108b10 con 0x7f227c1029e0 2024-03-16T12:19:01.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.183+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2264044070 con 0x7f227c1029e0 2024-03-16T12:19:01.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.185+0000 7f2273400640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 0x7f22600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.185+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f22640f07c0 con 0x7f227c1029e0 2024-03-16T12:19:01.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.185+0000 7f227aa00640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 0x7f22600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.186+0000 7f227aa00640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 0x7f22600a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f227c102840 tx=0x7f226c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:01.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.189+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22640bcf70 con 0x7f227c1029e0 2024-03-16T12:19:01.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:00 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1622249409' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:01.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:00 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2161467172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:01.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:00 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1622249409' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:01.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:00 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2161467172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:01.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:00 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1622249409' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:01.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:00 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2161467172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.386+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f227c062480 con 0x7f227c1029e0 2024-03-16T12:19:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.387+0000 7f2273400640 1 -- 172.21.15.98:0/1774695115 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f22640c1010 con 0x7f227c1029e0 2024-03-16T12:19:01.369 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:19:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.389+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 msgr2=0x7f22600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.389+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 0x7f22600a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f227c102840 tx=0x7f226c009500 comp rx=0 tx=0).stop 2024-03-16T12:19:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.389+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 msgr2=0x7f227c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.389+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f226402c8f0 tx=0x7f226402c660 comp rx=0 tx=0).stop 2024-03-16T12:19:01.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 shutdown_connections 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f22600a2880 0x7f22600a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f227c1033a0 0x7f227c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f227c1029e0 0x7f227c073bd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 --2- 172.21.15.98:0/1774695115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f227c1017f0 0x7f227c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 >> 172.21.15.98:0/1774695115 conn(0x7f227c0fcd60 msgr2=0x7f227c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 shutdown_connections 2024-03-16T12:19:01.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.390+0000 7f2281600640 1 -- 172.21.15.98:0/1774695115 wait complete. 2024-03-16T12:19:01.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 -- 172.21.15.98:0/2430561798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc074240 msgr2=0x7f1ffc0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 --2- 172.21.15.98:0/2430561798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc074240 0x7f1ffc0746c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009ca0 tx=0x7f1fec02c100 comp rx=0 tx=0).stop 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 -- 172.21.15.98:0/2430561798 shutdown_connections 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 --2- 172.21.15.98:0/2430561798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 --2- 172.21.15.98:0/2430561798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc074240 0x7f1ffc0746c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 --2- 172.21.15.98:0/2430561798 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc075820 0x7f1ffc073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 -- 172.21.15.98:0/2430561798 >> 172.21.15.98:0/2430561798 conn(0x7f1ffc0fcc50 msgr2=0x7f1ffc0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 -- 172.21.15.98:0/2430561798 shutdown_connections 2024-03-16T12:19:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.619+0000 7f2001600640 1 -- 172.21.15.98:0/2430561798 wait complete. 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.620+0000 7f2001600640 1 Processor -- start 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.620+0000 7f2001600640 1 -- start start 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 0x7f1ffc19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10f520 con 0x7f1ffc1095a0 2024-03-16T12:19:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10f3a0 con 0x7f1ffc074240 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2001600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10f6a0 con 0x7f1ffc075820 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2000c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 0x7f1ffc19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50142/0 (socket says 172.21.15.98:50142) 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffbe00640 1 -- 172.21.15.98:0/485145126 learned_addr learned my addr 172.21.15.98:0/485145126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 -- 172.21.15.98:0/485145126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 msgr2=0x7f1ffc199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 -- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 msgr2=0x7f1ffc1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffb400640 1 -- 172.21.15.98:0/485145126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1a17e0 con 0x7f1ffc075820 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f2000c00640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.621+0000 7f1ffbe00640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f1ffb400640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 0x7f1ffc19a100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009bf0 tx=0x7f1fec035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec039040 con 0x7f1ffc075820 2024-03-16T12:19:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec003970 con 0x7f1ffc075820 2024-03-16T12:19:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1a1a70 con 0x7f1ffc075820 2024-03-16T12:19:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec004360 con 0x7f1ffc075820 2024-03-16T12:19:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.622+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1a1f80 con 0x7f1ffc075820 2024-03-16T12:19:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.623+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc1082d0 con 0x7f1ffc075820 2024-03-16T12:19:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.625+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1fec046030 con 0x7f1ffc075820 2024-03-16T12:19:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.627+0000 7f1ff8c00640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.627+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1fec0f0cd0 con 0x7f1ffc075820 2024-03-16T12:19:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.627+0000 7f1ffbe00640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.628+0000 7f1ffbe00640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 0x7f1fe00a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4011550 tx=0x7f1fe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.629+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fec04d060 con 0x7f1ffc075820 2024-03-16T12:19:01.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.830+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1ffc02d340 con 0x7f1ffc075820 2024-03-16T12:19:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.831+0000 7f1ff8c00640 1 -- 172.21.15.98:0/485145126 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f1fec0bdb60 con 0x7f1ffc075820 2024-03-16T12:19:01.814 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:19:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 msgr2=0x7f1fe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 0x7f1fe00a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4011550 tx=0x7f1fe4005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 msgr2=0x7f1ffc19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 0x7f1ffc19a100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009bf0 tx=0x7f1fec035d10 comp rx=0 tx=0).stop 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 shutdown_connections 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc1095a0 0x7f1ffc1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc075820 0x7f1ffc19a100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 --2- 172.21.15.98:0/485145126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ffc074240 0x7f1ffc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.834+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 >> 172.21.15.98:0/485145126 conn(0x7f1ffc0fcc50 msgr2=0x7f1ffc1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.835+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 shutdown_connections 2024-03-16T12:19:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:01.835+0000 7f2001600640 1 -- 172.21.15.98:0/485145126 wait complete. 2024-03-16T12:19:01.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:19:02.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 -- 172.21.15.98:0/3541663055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa0104c10 msgr2=0x7f0aa0105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 --2- 172.21.15.98:0/3541663055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa0104c10 0x7f0aa0105090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009900 tx=0x7f0a8802bc50 comp rx=0 tx=0).stop 2024-03-16T12:19:02.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 -- 172.21.15.98:0/3541663055 shutdown_connections 2024-03-16T12:19:02.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 --2- 172.21.15.98:0/3541663055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa01055d0 0x7f0aa010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 --2- 172.21.15.98:0/3541663055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa0104c10 0x7f0aa0105090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 --2- 172.21.15.98:0/3541663055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0103a20 0x7f0aa0103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 -- 172.21.15.98:0/3541663055 >> 172.21.15.98:0/3541663055 conn(0x7f0aa00fef90 msgr2=0x7f0aa01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:02.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 -- 172.21.15.98:0/3541663055 shutdown_connections 2024-03-16T12:19:02.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.069+0000 7f0aa6600640 1 -- 172.21.15.98:0/3541663055 wait complete. 2024-03-16T12:19:02.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 Processor -- start 2024-03-16T12:19:02.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 -- start start 2024-03-16T12:19:02.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 0x7f0aa0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa01055d0 0x7f0aa01a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa010f6d0 con 0x7f0aa0103a20 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa010f550 con 0x7f0aa0104c10 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0aa6600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa010f850 con 0x7f0aa01055d0 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.071+0000 7f0a9f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 0x7f0aa0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50148/0 (socket says 172.21.15.98:50148) 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 -- 172.21.15.98:0/2432707568 learned_addr learned my addr 172.21.15.98:0/2432707568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a9f400640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 0x7f0aa0199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49858/0 (socket says 172.21.15.98:49858) 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a9fe00640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa01055d0 0x7f0aa01a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 -- 172.21.15.98:0/2432707568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa01055d0 msgr2=0x7f0aa01a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa01055d0 0x7f0aa01a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 -- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 msgr2=0x7f0aa0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 0x7f0aa0199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 -- 172.21.15.98:0/2432707568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa01a1950 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a97e00640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005ab0 tx=0x7f0a8802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.072+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88041070 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.073+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a88004450 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.073+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034440 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.073+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa01a1be0 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.073+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa01a20e0 con 0x7f0aa0104c10 2024-03-16T12:19:02.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.074+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa010ad40 con 0x7f0aa0104c10 2024-03-16T12:19:02.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.075+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0a88044070 con 0x7f0aa0104c10 2024-03-16T12:19:02.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.077+0000 7f0a9cc00640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 0x7f0a840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.077+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0a880f0920 con 0x7f0aa0104c10 2024-03-16T12:19:02.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.077+0000 7f0a9f400640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 0x7f0a840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.078+0000 7f0a9f400640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 0x7f0a840a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0aa019b210 tx=0x7f0a90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.081+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a880bd0d0 con 0x7f0aa0104c10 2024-03-16T12:19:02.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:01 smithi098 ceph-mon[31633]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:02.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:01 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3686869521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:01 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1774695115' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:01 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/485145126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:01 smithi153 ceph-mon[36901]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:02.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:01 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3686869521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:01 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1774695115' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:01 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/485145126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.276+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0aa002d3f0 con 0x7f0aa0104c10 2024-03-16T12:19:02.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.277+0000 7f0a9cc00640 1 -- 172.21.15.98:0/2432707568 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f0a880c1170 con 0x7f0aa0104c10 2024-03-16T12:19:02.260 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:19:02.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 msgr2=0x7f0a840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 0x7f0a840a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0aa019b210 tx=0x7f0a90005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 msgr2=0x7f0aa019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005ab0 tx=0x7f0a8802c6e0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 shutdown_connections 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0a840a2800 0x7f0a840a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa01055d0 0x7f0aa01a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa0104c10 0x7f0aa019a280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 --2- 172.21.15.98:0/2432707568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0aa0103a20 0x7f0aa0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 >> 172.21.15.98:0/2432707568 conn(0x7f0aa00fef90 msgr2=0x7f0aa0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 shutdown_connections 2024-03-16T12:19:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.280+0000 7f0aa6600640 1 -- 172.21.15.98:0/2432707568 wait complete. 2024-03-16T12:19:02.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T12:19:02.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:01 smithi173 ceph-mon[36632]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-16T12:19:02.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:01 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3686869521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:01 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1774695115' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:01 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/485145126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 -- 172.21.15.98:0/2521454386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 msgr2=0x7f7a88108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 --2- 172.21.15.98:0/2521454386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a88108220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009900 tx=0x7f7a7c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 -- 172.21.15.98:0/2521454386 shutdown_connections 2024-03-16T12:19:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 --2- 172.21.15.98:0/2521454386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a88110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 --2- 172.21.15.98:0/2521454386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a88109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 --2- 172.21.15.98:0/2521454386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a88108220 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 -- 172.21.15.98:0/2521454386 >> 172.21.15.98:0/2521454386 conn(0x7f7a88076da0 msgr2=0x7f7a880791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 -- 172.21.15.98:0/2521454386 shutdown_connections 2024-03-16T12:19:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.505+0000 7f7a90800640 1 -- 172.21.15.98:0/2521454386 wait complete. 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.506+0000 7f7a90800640 1 Processor -- start 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.506+0000 7f7a90800640 1 -- start start 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a8819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a881a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a88113a50 con 0x7f7a88108ff0 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881138d0 con 0x7f7a88107e00 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a90800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7a88113bd0 con 0x7f7a881099b0 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a8819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50162/0 (socket says 172.21.15.98:50162) 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 -- 172.21.15.98:0/2625081946 learned_addr learned my addr 172.21.15.98:0/2625081946 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8e800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a881a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 -- 172.21.15.98:0/2625081946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 msgr2=0x7f7a881a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a881a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 -- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 msgr2=0x7f7a8819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a8819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8de00640 1 -- 172.21.15.98:0/2625081946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a881a5d40 con 0x7f7a88107e00 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8e800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a881a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.507+0000 7f7a8d400640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a8819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a8de00640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c000c00 tx=0x7f7a7c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c041070 con 0x7f7a88107e00 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a7c0044d0 con 0x7f7a88107e00 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a881a5fd0 con 0x7f7a88107e00 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c034440 con 0x7f7a88107e00 2024-03-16T12:19:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.508+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a881a63e0 con 0x7f7a88107e00 2024-03-16T12:19:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.509+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8810f1f0 con 0x7f7a88107e00 2024-03-16T12:19:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.511+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7a7c044070 con 0x7f7a88107e00 2024-03-16T12:19:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.512+0000 7f7a82a00640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 0x7f7a700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.512+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7a7c0f0a90 con 0x7f7a88107e00 2024-03-16T12:19:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.512+0000 7f7a8d400640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 0x7f7a700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.513+0000 7f7a8d400640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 0x7f7a700a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a8819f630 tx=0x7f7a78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.514+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a7c0bd890 con 0x7f7a88107e00 2024-03-16T12:19:02.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.728+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7a8802d1b0 con 0x7f7a88107e00 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.728+0000 7f7a82a00640 1 -- 172.21.15.98:0/2625081946 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+588 (secure 0 0 0) 0x7f7a7c0bd190 con 0x7f7a88107e00 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/352508062 2024-03-17T12:10:42.632912+0000 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/653650228 2024-03-17T12:10:08.286891+0000 2024-03-16T12:19:02.711 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/719012091 2024-03-17T12:10:42.632912+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/196846375 2024-03-17T12:10:42.632912+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3496387635 2024-03-17T12:10:08.286891+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3016831032 2024-03-17T12:10:08.286891+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:19:02.712 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-16T12:19:02.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.731+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 msgr2=0x7f7a700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.731+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 0x7f7a700a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a8819f630 tx=0x7f7a78005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:02.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.731+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 msgr2=0x7f7a8819e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:02.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.731+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c000c00 tx=0x7f7a7c005d50 comp rx=0 tx=0).stop 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 shutdown_connections 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7a700a2800 0x7f7a700a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a881099b0 0x7f7a881a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a88108ff0 0x7f7a8819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 --2- 172.21.15.98:0/2625081946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a88107e00 0x7f7a8819e120 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 >> 172.21.15.98:0/2625081946 conn(0x7f7a88076da0 msgr2=0x7f7a8810c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 shutdown_connections 2024-03-16T12:19:02.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:02.732+0000 7f7a90800640 1 -- 172.21.15.98:0/2625081946 wait complete. 2024-03-16T12:19:02.733 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-03-16T12:19:02.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:19:02.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-16T12:19:02.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[75793]: fuse finished with error 0 and tester_r 0 2024-03-16T12:19:02.821 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T12:19:02.907 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:19:02.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:19:02.908 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-16T12:19:02.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:19:02.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:02 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2432707568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:02.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:02 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2625081946' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 -- 172.21.15.98:0/2017216039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f81099b0 msgr2=0x7f34f8110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 --2- 172.21.15.98:0/2017216039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f81099b0 0x7f34f8110240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009ca0 tx=0x7f34f002c100 comp rx=0 tx=0).stop 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 -- 172.21.15.98:0/2017216039 shutdown_connections 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 --2- 172.21.15.98:0/2017216039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f81099b0 0x7f34f8110240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 --2- 172.21.15.98:0/2017216039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 0x7f34f8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 --2- 172.21.15.98:0/2017216039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f8107e00 0x7f34f8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.195+0000 7f34ffc00640 1 -- 172.21.15.98:0/2017216039 >> 172.21.15.98:0/2017216039 conn(0x7f34f8076dc0 msgr2=0x7f34f80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.196+0000 7f34ffc00640 1 -- 172.21.15.98:0/2017216039 shutdown_connections 2024-03-16T12:19:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.196+0000 7f34ffc00640 1 -- 172.21.15.98:0/2017216039 wait complete. 2024-03-16T12:19:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.196+0000 7f34ffc00640 1 Processor -- start 2024-03-16T12:19:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 -- start start 2024-03-16T12:19:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 0x7f34f819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 0x7f34f81a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34f8113a80 con 0x7f34f81099b0 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34f8113900 con 0x7f34f8107e00 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34ffc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f34f8113c00 con 0x7f34f8108ff0 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34fd200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34fdc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 0x7f34f81a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34fd200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50182/0 (socket says 172.21.15.98:50182) 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34fd200640 1 -- 172.21.15.98:0/3332791081 learned_addr learned my addr 172.21.15.98:0/3332791081 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.197+0000 7f34efe00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 0x7f34f819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 -- 172.21.15.98:0/3332791081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 msgr2=0x7f34f819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 0x7f34f819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 -- 172.21.15.98:0/3332791081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 msgr2=0x7f34f819e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 -- 172.21.15.98:0/3332791081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f81a5cb0 con 0x7f34f81099b0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fd200640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34fdc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 0x7f34f81a5640 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34f0037040 tx=0x7f34f0035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0039040 con 0x7f34f81099b0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0044070 con 0x7f34f81099b0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f81a5f40 con 0x7f34f81099b0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.198+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0004350 con 0x7f34f81099b0 2024-03-16T12:19:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.199+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f81a63a0 con 0x7f34f81099b0 2024-03-16T12:19:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.200+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f810f120 con 0x7f34f81099b0 2024-03-16T12:19:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.202+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f34f00044f0 con 0x7f34f81099b0 2024-03-16T12:19:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.203+0000 7f34f6000640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.203+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f34f00f1cc0 con 0x7f34f81099b0 2024-03-16T12:19:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.203+0000 7f34fd200640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.204+0000 7f34fd200640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 0x7f34d80a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34e00058c0 tx=0x7f34e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.206+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f34f00beac0 con 0x7f34f81099b0 2024-03-16T12:19:03.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:02 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2432707568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:03.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:02 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2625081946' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:03.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:02 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2432707568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:19:03.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:02 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2625081946' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:19:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.404+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34f802d330 con 0x7f34f81099b0 2024-03-16T12:19:03.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.407+0000 7f34f6000640 1 -- 172.21.15.98:0/3332791081 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7f34f00098b0 con 0x7f34f81099b0 2024-03-16T12:19:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:03.391 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.410+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 msgr2=0x7f34d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.410+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 0x7f34d80a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34e00058c0 tx=0x7f34e0005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.410+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 msgr2=0x7f34f81a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.410+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 0x7f34f81a5640 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34f0037040 tx=0x7f34f0035d10 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 shutdown_connections 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f34d80a2800 0x7f34d80a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f81099b0 0x7f34f81a5640 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34f8108ff0 0x7f34f819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 --2- 172.21.15.98:0/3332791081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34f8107e00 0x7f34f819e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 >> 172.21.15.98:0/3332791081 conn(0x7f34f8076dc0 msgr2=0x7f34f810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 shutdown_connections 2024-03-16T12:19:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.411+0000 7f34ffc00640 1 -- 172.21.15.98:0/3332791081 wait complete. 2024-03-16T12:19:03.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:19:03.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.688+0000 7faea4200640 1 -- 172.21.15.98:0/2061199746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c108ff0 msgr2=0x7fae9c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.688+0000 7faea4200640 1 --2- 172.21.15.98:0/2061199746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c108ff0 0x7fae9c109470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fae88009900 tx=0x7fae8802bc50 comp rx=0 tx=0).stop 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 -- 172.21.15.98:0/2061199746 shutdown_connections 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 --2- 172.21.15.98:0/2061199746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c1099b0 0x7fae9c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 --2- 172.21.15.98:0/2061199746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c108ff0 0x7fae9c109470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 --2- 172.21.15.98:0/2061199746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c107e00 0x7fae9c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 -- 172.21.15.98:0/2061199746 >> 172.21.15.98:0/2061199746 conn(0x7fae9c076dc0 msgr2=0x7fae9c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 -- 172.21.15.98:0/2061199746 shutdown_connections 2024-03-16T12:19:03.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.689+0000 7faea4200640 1 -- 172.21.15.98:0/2061199746 wait complete. 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 Processor -- start 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 -- start start 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 0x7fae9c19e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 0x7fae9c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c113ab0 con 0x7fae9c107e00 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c113930 con 0x7fae9c1099b0 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea4200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c113c30 con 0x7fae9c108ff0 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 0x7fae9c19e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea2200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea2200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50198/0 (socket says 172.21.15.98:50198) 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.690+0000 7faea2200640 1 -- 172.21.15.98:0/1874736279 learned_addr learned my addr 172.21.15.98:0/1874736279 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea0e00640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 0x7fae9c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 -- 172.21.15.98:0/1874736279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 msgr2=0x7fae9c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 0x7fae9c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 -- 172.21.15.98:0/1874736279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 msgr2=0x7fae9c1a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 -- 172.21.15.98:0/1874736279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9c1a5d10 con 0x7fae9c107e00 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea0e00640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 0x7fae9c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:19:03.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea2200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea1800640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 0x7fae9c19e0f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fae9000e9c0 tx=0x7fae9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9000cd80 con 0x7fae9c107e00 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae90010910 con 0x7fae9c107e00 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9c1a5fa0 con 0x7fae9c107e00 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.691+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90018980 con 0x7fae9c107e00 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.692+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9c1a6460 con 0x7fae9c107e00 2024-03-16T12:19:03.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.693+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9c10f0c0 con 0x7fae9c107e00 2024-03-16T12:19:03.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.695+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fae9001f020 con 0x7fae9c107e00 2024-03-16T12:19:03.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.696+0000 7fae9a000640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 0x7fae840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:03.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.696+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fae900ced60 con 0x7fae9c107e00 2024-03-16T12:19:03.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.696+0000 7faea0e00640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 0x7fae840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:03.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.697+0000 7faea0e00640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 0x7fae840a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae8802c260 tx=0x7fae88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:03.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.700+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae9009bb60 con 0x7fae9c107e00 2024-03-16T12:19:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.898+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae9c062410 con 0x7fae9c107e00 2024-03-16T12:19:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.900+0000 7fae9a000640 1 -- 172.21.15.98:0/1874736279 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fae9009b460 con 0x7fae9c107e00 2024-03-16T12:19:03.883 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:03.883 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.902+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 msgr2=0x7fae840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.902+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 0x7fae840a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae8802c260 tx=0x7fae88005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 msgr2=0x7fae9c19e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 0x7fae9c19e0f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fae9000e9c0 tx=0x7fae9000ee80 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 shutdown_connections 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fae840a2800 0x7fae840a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c1099b0 0x7fae9c1a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae9c108ff0 0x7fae9c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 --2- 172.21.15.98:0/1874736279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae9c107e00 0x7fae9c19e0f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 >> 172.21.15.98:0/1874736279 conn(0x7fae9c076dc0 msgr2=0x7fae9c10c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 shutdown_connections 2024-03-16T12:19:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:03.903+0000 7faea4200640 1 -- 172.21.15.98:0/1874736279 wait complete. 2024-03-16T12:19:03.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 -- 172.21.15.98:0/1292513138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6801055d0 msgr2=0x7fb68010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 --2- 172.21.15.98:0/1292513138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6801055d0 0x7fb68010be60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb678009ca0 tx=0x7fb67802c0f0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 -- 172.21.15.98:0/1292513138 shutdown_connections 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 --2- 172.21.15.98:0/1292513138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6801055d0 0x7fb68010be60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 --2- 172.21.15.98:0/1292513138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 0x7fb680105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 --2- 172.21.15.98:0/1292513138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb680103a20 0x7fb680103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 -- 172.21.15.98:0/1292513138 >> 172.21.15.98:0/1292513138 conn(0x7fb6800fef90 msgr2=0x7fb6801013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 -- 172.21.15.98:0/1292513138 shutdown_connections 2024-03-16T12:19:04.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.126+0000 7fb688400640 1 -- 172.21.15.98:0/1292513138 wait complete. 2024-03-16T12:19:04.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 Processor -- start 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 -- start start 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 0x7fb68019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 0x7fb6801a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb68010f6a0 con 0x7fb6801055d0 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb68010f520 con 0x7fb680103a20 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.127+0000 7fb688400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb68010f820 con 0x7fb680104c10 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50224/0 (socket says 172.21.15.98:50224) 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 -- 172.21.15.98:0/551762118 learned_addr learned my addr 172.21.15.98:0/551762118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb686400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 0x7fb6801a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685000640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 0x7fb68019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 -- 172.21.15.98:0/551762118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 msgr2=0x7fb68019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 0x7fb68019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 -- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 msgr2=0x7fb6801a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 0x7fb6801a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 -- 172.21.15.98:0/551762118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6801a1940 con 0x7fb680103a20 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb686400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 0x7fb6801a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb685a00640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb67400e9c0 tx=0x7fb67400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67400cd80 con 0x7fb680103a20 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.128+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6740035f0 con 0x7fb680103a20 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.129+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6801a1c30 con 0x7fb680103a20 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.129+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674010690 con 0x7fb680103a20 2024-03-16T12:19:04.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.129+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6801a21d0 con 0x7fb680103a20 2024-03-16T12:19:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.130+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb68010adc0 con 0x7fb680103a20 2024-03-16T12:19:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.131+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb6740037a0 con 0x7fb680103a20 2024-03-16T12:19:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.133+0000 7fb67e000640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 0x7fb6680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.133+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb6740cf170 con 0x7fb680103a20 2024-03-16T12:19:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.133+0000 7fb685000640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 0x7fb6680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:04.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.133+0000 7fb685000640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 0x7fb6680a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009bd0 tx=0x7fb66c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:04.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.137+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb67409bf70 con 0x7fb680103a20 2024-03-16T12:19:04.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:03 smithi153 ceph-mon[36901]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:04.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:03 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3332791081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:04.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:03 smithi098 ceph-mon[31633]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:04.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:03 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3332791081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:04.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:03 smithi173 ceph-mon[36632]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:04.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:03 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3332791081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.373+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb68002d3f0 con 0x7fb680103a20 2024-03-16T12:19:04.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.887+0000 7fb67e000640 1 -- 172.21.15.98:0/551762118 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb67409b870 con 0x7fb680103a20 2024-03-16T12:19:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.890+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 msgr2=0x7fb6680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.890+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 0x7fb6680a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009bd0 tx=0x7fb66c005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.890+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 msgr2=0x7fb680199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb67400e9c0 tx=0x7fb67400ee80 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 shutdown_connections 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb6680a2880 0x7fb6680a4d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6801055d0 0x7fb6801a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb680104c10 0x7fb68019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 --2- 172.21.15.98:0/551762118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb680103a20 0x7fb680199d30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 >> 172.21.15.98:0/551762118 conn(0x7fb6800fef90 msgr2=0x7fb680107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 shutdown_connections 2024-03-16T12:19:04.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:04.891+0000 7fb688400640 1 -- 172.21.15.98:0/551762118 wait complete. 2024-03-16T12:19:04.893 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-16T12:19:04.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 -- 172.21.15.98:0/2203628202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc100ae0 msgr2=0x7f61bc100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 --2- 172.21.15.98:0/2203628202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc100ae0 0x7f61bc100f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009900 tx=0x7f61b002bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 -- 172.21.15.98:0/2203628202 shutdown_connections 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 --2- 172.21.15.98:0/2203628202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc1014a0 0x7f61bc107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 --2- 172.21.15.98:0/2203628202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc100ae0 0x7f61bc100f60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 --2- 172.21.15.98:0/2203628202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 -- 172.21.15.98:0/2203628202 >> 172.21.15.98:0/2203628202 conn(0x7f61bc0fae60 msgr2=0x7f61bc0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 -- 172.21.15.98:0/2203628202 shutdown_connections 2024-03-16T12:19:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.126+0000 7f61c2e00640 1 -- 172.21.15.98:0/2203628202 wait complete. 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.127+0000 7f61c2e00640 1 Processor -- start 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.127+0000 7f61c2e00640 1 -- start start 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc100ae0 0x7f61bc101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 0x7f61bc102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc10b140 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc10afc0 con 0x7f61bc1014a0 2024-03-16T12:19:05.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc10b2c0 con 0x7f61bc100ae0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 0x7f61bc102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49940/0 (socket says 172.21.15.98:49940) 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 -- 172.21.15.98:0/30534092 learned_addr learned my addr 172.21.15.98:0/30534092 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1000640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc100ae0 0x7f61bc101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 -- 172.21.15.98:0/30534092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc100ae0 msgr2=0x7f61bc101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc100ae0 0x7f61bc101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 -- 172.21.15.98:0/30534092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 msgr2=0x7f61bc102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 0x7f61bc102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c1a00640 1 -- 172.21.15.98:0/30534092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc102c90 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.128+0000 7f61c2400640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 0x7f61bc102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61c1a00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61a80097d0 tx=0x7f61a8009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8018030 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc1a1ad0 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a800e8c0 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8003890 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.129+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc1a2040 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.130+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61bc1a2210 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.133+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f61a801b070 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.134+0000 7f61ba000640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 0x7f61a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.134+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61a80ce3f0 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.134+0000 7f61c1000640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 0x7f61a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:05.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.135+0000 7f61c1000640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 0x7f61a40a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61b002c7b0 tx=0x7f61b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:05.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.139+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61a8022070 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:04 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1874736279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:05.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:04 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/551762118' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:04 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:04 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1874736279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:05.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:04 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/551762118' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:04 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:04 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1874736279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:05.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:04 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/551762118' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:04 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T12:19:05.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.377+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f61bc1a23e0 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.886+0000 7f61ba000640 1 -- 172.21.15.98:0/30534092 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f61a8011070 con 0x7f61bc0ff8f0 2024-03-16T12:19:05.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.890+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 msgr2=0x7f61a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:05.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.890+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 0x7f61a40a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61b002c7b0 tx=0x7f61b0005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.890+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 msgr2=0x7f61bc103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:05.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.890+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61a80097d0 tx=0x7f61a8009c90 comp rx=0 tx=0).stop 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 shutdown_connections 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61a40a2880 0x7f61a40a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61bc1014a0 0x7f61bc102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61bc100ae0 0x7f61bc101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 --2- 172.21.15.98:0/30534092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61bc0ff8f0 0x7f61bc103880 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 >> 172.21.15.98:0/30534092 conn(0x7f61bc0fae60 msgr2=0x7f61bc105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 shutdown_connections 2024-03-16T12:19:05.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:05.891+0000 7f61c2e00640 1 -- 172.21.15.98:0/30534092 wait complete. 2024-03-16T12:19:05.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 -- 172.21.15.98:0/2755871893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e480773e0 msgr2=0x7f2e48077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2755871893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e480773e0 0x7f2e48077840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009900 tx=0x7f2e3002bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 -- 172.21.15.98:0/2755871893 shutdown_connections 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2755871893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e48077d80 0x7f2e48078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2755871893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e480773e0 0x7f2e48077840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2755871893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480761a0 0x7f2e480765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 -- 172.21.15.98:0/2755871893 >> 172.21.15.98:0/2755871893 conn(0x7f2e480fef90 msgr2=0x7f2e481013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:06.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 -- 172.21.15.98:0/2755871893 shutdown_connections 2024-03-16T12:19:06.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.119+0000 7f2e4e000640 1 -- 172.21.15.98:0/2755871893 wait complete. 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.120+0000 7f2e4e000640 1 Processor -- start 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 -- start start 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 0x7f2e4819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 0x7f2e481a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48113af0 con 0x7f2e480761a0 2024-03-16T12:19:06.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48113970 con 0x7f2e48077d80 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e4e000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48113c70 con 0x7f2e480773e0 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e47400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e47400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49958/0 (socket says 172.21.15.98:49958) 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e47e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 0x7f2e481a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.121+0000 7f2e47400640 1 -- 172.21.15.98:0/2897522128 learned_addr learned my addr 172.21.15.98:0/2897522128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e46a00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 0x7f2e4819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 -- 172.21.15.98:0/2897522128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 msgr2=0x7f2e4819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 0x7f2e4819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 -- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 msgr2=0x7f2e4819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 -- 172.21.15.98:0/2897522128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e481a5d30 con 0x7f2e48077d80 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e46a00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 0x7f2e4819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47400640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e47e00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 0x7f2e481a56c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800e970 tx=0x7f2e3800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800ccc0 con 0x7f2e48077d80 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380035f0 con 0x7f2e48077d80 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e481a5fc0 con 0x7f2e48077d80 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.122+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800ce60 con 0x7f2e48077d80 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.123+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e481a6490 con 0x7f2e48077d80 2024-03-16T12:19:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.124+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e48110390 con 0x7f2e48077d80 2024-03-16T12:19:06.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.126+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2e380037a0 con 0x7f2e48077d80 2024-03-16T12:19:06.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.127+0000 7f2e3fe00640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 0x7f2e2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:06.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.127+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2e380cef50 con 0x7f2e48077d80 2024-03-16T12:19:06.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.127+0000 7f2e47400640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 0x7f2e2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:06.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.128+0000 7f2e47400640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 0x7f2e2c0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e400049d0 tx=0x7f2e40009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:06.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.129+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e3809bd50 con 0x7f2e48077d80 2024-03-16T12:19:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:05 smithi098 ceph-mon[31633]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:05 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T12:19:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:05 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:06.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:05 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T12:19:06.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:05 smithi153 ceph-mon[36901]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:06.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:05 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T12:19:06.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:05 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:06.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:05 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T12:19:06.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:05 smithi173 ceph-mon[36632]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:06.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:05 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T12:19:06.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:05 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:06.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:05 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T12:19:06.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.372+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f2e4802d1b0 con 0x7f2e48077d80 2024-03-16T12:19:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.910+0000 7f2e3fe00640 1 -- 172.21.15.98:0/2897522128 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f2e3809b650 con 0x7f2e48077d80 2024-03-16T12:19:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 msgr2=0x7f2e2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 0x7f2e2c0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e400049d0 tx=0x7f2e40009500 comp rx=0 tx=0).stop 2024-03-16T12:19:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 msgr2=0x7f2e481a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 0x7f2e481a56c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800e970 tx=0x7f2e3800ee30 comp rx=0 tx=0).stop 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 shutdown_connections 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2e2c0a2800 0x7f2e2c0a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e48077d80 0x7f2e481a56c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e480773e0 0x7f2e4819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 --2- 172.21.15.98:0/2897522128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e480761a0 0x7f2e4819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.913+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 >> 172.21.15.98:0/2897522128 conn(0x7f2e480fef90 msgr2=0x7f2e48100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.914+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 shutdown_connections 2024-03-16T12:19:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:06.914+0000 7f2e4e000640 1 -- 172.21.15.98:0/2897522128 wait complete. 2024-03-16T12:19:06.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 -- 172.21.15.98:0/2957150558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 msgr2=0x7fce54109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 --2- 172.21.15.98:0/2957150558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce54109370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce44009900 tx=0x7fce4402bc40 comp rx=0 tx=0).stop 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 -- 172.21.15.98:0/2957150558 shutdown_connections 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 --2- 172.21.15.98:0/2957150558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce541098b0 0x7fce54110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 --2- 172.21.15.98:0/2957150558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce54109370 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 --2- 172.21.15.98:0/2957150558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce54107d00 0x7fce54108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 -- 172.21.15.98:0/2957150558 >> 172.21.15.98:0/2957150558 conn(0x7fce54076cb0 msgr2=0x7fce540790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 -- 172.21.15.98:0/2957150558 shutdown_connections 2024-03-16T12:19:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.136+0000 7fce5b200640 1 -- 172.21.15.98:0/2957150558 wait complete. 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.137+0000 7fce5b200640 1 Processor -- start 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.137+0000 7fce5b200640 1 -- start start 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 0x7fce5419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce541098b0 0x7fce541a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce541139d0 con 0x7fce54107d00 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce54113850 con 0x7fce54108ef0 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce54113b50 con 0x7fce541098b0 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 0x7fce5419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:47740/0 (socket says 172.21.15.98:47740) 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 -- 172.21.15.98:0/3389454588 learned_addr learned my addr 172.21.15.98:0/3389454588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce5a800640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce541098b0 0x7fce541a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 -- 172.21.15.98:0/3389454588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce541098b0 msgr2=0x7fce541a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce541098b0 0x7fce541a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 -- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 msgr2=0x7fce5419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.138+0000 7fce59400640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 0x7fce5419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce59400640 1 -- 172.21.15.98:0/3389454588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce541a5c00 con 0x7fce54108ef0 2024-03-16T12:19:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce59e00640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 0x7fce5419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce59400640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce4402c8f0 tx=0x7fce44005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44041070 con 0x7fce54108ef0 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce440044d0 con 0x7fce54108ef0 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce541a5e90 con 0x7fce54108ef0 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44034440 con 0x7fce54108ef0 2024-03-16T12:19:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.139+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce541a6300 con 0x7fce54108ef0 2024-03-16T12:19:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.140+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5410efc0 con 0x7fce54108ef0 2024-03-16T12:19:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.142+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fce44044070 con 0x7fce54108ef0 2024-03-16T12:19:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.143+0000 7fce4ea00640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 0x7fce3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.144+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fce440f07e0 con 0x7fce54108ef0 2024-03-16T12:19:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.144+0000 7fce59e00640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 0x7fce3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.144+0000 7fce59e00640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 0x7fce3c0a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce54108d50 tx=0x7fce48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.147+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce440bcf90 con 0x7fce54108ef0 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2897522128' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:06 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T12:19:07.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T12:19:07.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T12:19:07.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:07.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T12:19:07.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T12:19:07.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2897522128' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:06 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/30534092' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2897522128' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:06 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T12:19:07.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.388+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fce540623e0 con 0x7fce54108ef0 2024-03-16T12:19:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.483+0000 7fce4ea00640 1 -- 172.21.15.98:0/3389454588 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fce440c1030 con 0x7fce54108ef0 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.487+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 msgr2=0x7fce3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 0x7fce3c0a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce54108d50 tx=0x7fce48005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 msgr2=0x7fce5419e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce4402c8f0 tx=0x7fce44005d50 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 shutdown_connections 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce3c0a2880 0x7fce3c0a4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce541098b0 0x7fce541a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce54108ef0 0x7fce5419e520 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 --2- 172.21.15.98:0/3389454588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce54107d00 0x7fce5419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 >> 172.21.15.98:0/3389454588 conn(0x7fce54076cb0 msgr2=0x7fce5410c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 shutdown_connections 2024-03-16T12:19:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.488+0000 7fce5b200640 1 -- 172.21.15.98:0/3389454588 wait complete. 2024-03-16T12:19:07.491 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-16T12:19:07.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T12:19:07.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 -- 172.21.15.98:0/1684197976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 msgr2=0x7f98cc108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 --2- 172.21.15.98:0/1684197976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc108220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009900 tx=0x7f98bc02bc50 comp rx=0 tx=0).stop 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 -- 172.21.15.98:0/1684197976 shutdown_connections 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 --2- 172.21.15.98:0/1684197976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 0x7f98cc110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 --2- 172.21.15.98:0/1684197976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 0x7f98cc109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 --2- 172.21.15.98:0/1684197976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc108220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 -- 172.21.15.98:0/1684197976 >> 172.21.15.98:0/1684197976 conn(0x7f98cc076da0 msgr2=0x7f98cc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 -- 172.21.15.98:0/1684197976 shutdown_connections 2024-03-16T12:19:07.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.722+0000 7f98d2000640 1 -- 172.21.15.98:0/1684197976 wait complete. 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 Processor -- start 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 -- start start 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 0x7f98cc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 0x7f98cc1a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc113aa0 con 0x7f98cc108ff0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc113920 con 0x7f98cc1099b0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.723+0000 7f98d2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc113c20 con 0x7f98cc107e00 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50686/0 (socket says 172.21.15.98:50686) 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 -- 172.21.15.98:0/2209517746 learned_addr learned my addr 172.21.15.98:0/2209517746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cbe00640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 0x7f98cc1a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98caa00640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 0x7f98cc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 -- 172.21.15.98:0/2209517746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 msgr2=0x7f98cc1a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 0x7f98cc1a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 -- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 msgr2=0x7f98cc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 0x7f98cc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 -- 172.21.15.98:0/2209517746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98cc1a5d10 con 0x7f98cc107e00 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.724+0000 7f98cb400640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98bc02c260 tx=0x7f98bc02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.725+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc041070 con 0x7f98cc107e00 2024-03-16T12:19:07.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.725+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98bc004450 con 0x7f98cc107e00 2024-03-16T12:19:07.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.725+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc034440 con 0x7f98cc107e00 2024-03-16T12:19:07.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.725+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98cc1a5fa0 con 0x7f98cc107e00 2024-03-16T12:19:07.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.725+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98cc1a63b0 con 0x7f98cc107e00 2024-03-16T12:19:07.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.726+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98cc10f120 con 0x7f98cc107e00 2024-03-16T12:19:07.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.728+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f98bc044070 con 0x7f98cc107e00 2024-03-16T12:19:07.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.729+0000 7f98c3e00640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:07.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.729+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98bc0f07c0 con 0x7f98cc107e00 2024-03-16T12:19:07.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.729+0000 7f98caa00640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:07.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.730+0000 7f98caa00640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 0x7f98b00a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98cc19f600 tx=0x7f98b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:07.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.733+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98bc0bcf70 con 0x7f98cc107e00 2024-03-16T12:19:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.961+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f98cc061c20 con 0x7f98b00a2800 2024-03-16T12:19:07.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.990+0000 7f98c3e00640 1 -- 172.21.15.98:0/2209517746 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f98cc061c20 con 0x7f98b00a2800 2024-03-16T12:19:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 msgr2=0x7f98b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 0x7f98b00a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98cc19f600 tx=0x7f98b8005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 msgr2=0x7f98cc19e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98bc02c260 tx=0x7f98bc02c670 comp rx=0 tx=0).stop 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 shutdown_connections 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f98b00a2800 0x7f98b00a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98cc1099b0 0x7f98cc1a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98cc108ff0 0x7f98cc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 --2- 172.21.15.98:0/2209517746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98cc107e00 0x7f98cc19e0f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.993+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 >> 172.21.15.98:0/2209517746 conn(0x7f98cc076da0 msgr2=0x7f98cc10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.994+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 shutdown_connections 2024-03-16T12:19:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:07.994+0000 7f98d2000640 1 -- 172.21.15.98:0/2209517746 wait complete. 2024-03-16T12:19:07.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T12:19:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1722906058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 msgr2=0x7fb4dc109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1722906058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc109370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009860 tx=0x7fb4c402bb70 comp rx=0 tx=0).stop 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1722906058 shutdown_connections 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1722906058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1722906058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc109370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1722906058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1722906058 >> 172.21.15.98:0/1722906058 conn(0x7fb4dc076cb0 msgr2=0x7fb4dc0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1722906058 shutdown_connections 2024-03-16T12:19:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.217+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1722906058 wait complete. 2024-03-16T12:19:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 Processor -- start 2024-03-16T12:19:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 -- start start 2024-03-16T12:19:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc113960 con 0x7fb4dc1098b0 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc1137e0 con 0x7fb4dc107d00 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.218+0000 7fb4e0e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc113ae0 con 0x7fb4dc108ef0 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47040/0 (socket says 172.21.15.98:47040) 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4dbe00640 1 -- 172.21.15.98:0/1669942025 learned_addr learned my addr 172.21.15.98:0/1669942025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4d2a00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 -- 172.21.15.98:0/1669942025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 msgr2=0x7fb4dc19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 -- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 msgr2=0x7fb4dc1a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 -- 172.21.15.98:0/1669942025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4dc1a5bb0 con 0x7fb4dc107d00 2024-03-16T12:19:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4d2a00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4dbe00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.219+0000 7fb4db400640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc19e030 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc108d50 tx=0x7fb4cc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.220+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc01b070 con 0x7fb4dc107d00 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.220+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4cc003b80 con 0x7fb4dc107d00 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.220+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4dc1a5ea0 con 0x7fb4dc107d00 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.220+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0168d0 con 0x7fb4dc107d00 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.220+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4dc1a6310 con 0x7fb4dc107d00 2024-03-16T12:19:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.221+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4dc10f020 con 0x7fb4dc107d00 2024-03-16T12:19:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.223+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb4cc01e070 con 0x7fb4dc107d00 2024-03-16T12:19:08.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.224+0000 7fb4d8c00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 0x7fb4c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.224+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb4cc0cea80 con 0x7fb4dc107d00 2024-03-16T12:19:08.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.224+0000 7fb4d2a00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 0x7fb4c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.225+0000 7fb4d2a00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 0x7fb4c00a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c180 tx=0x7fb4c402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.231+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb4cc09b230 con 0x7fb4dc107d00 2024-03-16T12:19:08.243 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:19:07.972+0000 7fcea9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:19:07.972+0000 7fcea9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:19:07.972+0000 7fcea9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:19:07.972+0000 7fcea9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:19:07.972+0000 7fcea9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:active 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:19:08.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3389454588' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:active 2024-03-16T12:19:08.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:07 smithi098 ceph-mon[31633]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:19:08.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:active 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3389454588' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:active 2024-03-16T12:19:08.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:07 smithi153 ceph-mon[36901]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] up:active 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3389454588' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] up:active 2024-03-16T12:19:08.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:07 smithi173 ceph-mon[36632]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:19:08.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.464+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb4dc061cb0 con 0x7fb4c00a2880 2024-03-16T12:19:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.495+0000 7fb4d8c00640 1 -- 172.21.15.98:0/1669942025 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb4dc061cb0 con 0x7fb4c00a2880 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 msgr2=0x7fb4c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 0x7fb4c00a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c180 tx=0x7fb4c402e040 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 msgr2=0x7fb4dc19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc19e030 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc108d50 tx=0x7fb4cc00c800 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 shutdown_connections 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb4c00a2880 0x7fb4c00a4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4dc1098b0 0x7fb4dc1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4dc108ef0 0x7fb4dc19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 --2- 172.21.15.98:0/1669942025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4dc107d00 0x7fb4dc19e030 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 >> 172.21.15.98:0/1669942025 conn(0x7fb4dc076cb0 msgr2=0x7fb4dc10c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 shutdown_connections 2024-03-16T12:19:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.500+0000 7fb4e0e00640 1 -- 172.21.15.98:0/1669942025 wait complete. 2024-03-16T12:19:08.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 -- 172.21.15.98:0/2568703259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c108ff0 msgr2=0x7f4a3c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 --2- 172.21.15.98:0/2568703259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c109470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009900 tx=0x7f4a2c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 -- 172.21.15.98:0/2568703259 shutdown_connections 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 --2- 172.21.15.98:0/2568703259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 0x7f4a3c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 --2- 172.21.15.98:0/2568703259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c109470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 --2- 172.21.15.98:0/2568703259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c107e00 0x7f4a3c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 -- 172.21.15.98:0/2568703259 >> 172.21.15.98:0/2568703259 conn(0x7f4a3c076da0 msgr2=0x7f4a3c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 -- 172.21.15.98:0/2568703259 shutdown_connections 2024-03-16T12:19:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.736+0000 7f4a43e00640 1 -- 172.21.15.98:0/2568703259 wait complete. 2024-03-16T12:19:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.737+0000 7f4a43e00640 1 Processor -- start 2024-03-16T12:19:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.737+0000 7f4a43e00640 1 -- start start 2024-03-16T12:19:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 0x7f4a3c1a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113a80 con 0x7f4a3c1099b0 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113900 con 0x7f4a3c108ff0 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a43e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113c00 con 0x7f4a3c107e00 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50752/0 (socket says 172.21.15.98:50752) 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 -- 172.21.15.98:0/657231039 learned_addr learned my addr 172.21.15.98:0/657231039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 0x7f4a3c1a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a40a00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 -- 172.21.15.98:0/657231039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c108ff0 msgr2=0x7f4a3c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 -- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 msgr2=0x7f4a3c1a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 0x7f4a3c1a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.738+0000 7f4a41400640 1 -- 172.21.15.98:0/657231039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1a5d40 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a41400640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e9c0 tx=0x7f4a3000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000cd80 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a300035f0 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c1a5fd0 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30010690 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.739+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c1a6440 con 0x7f4a3c107e00 2024-03-16T12:19:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.740+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c10f1a0 con 0x7f4a3c107e00 2024-03-16T12:19:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.742+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4a300037a0 con 0x7f4a3c107e00 2024-03-16T12:19:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.743+0000 7f4a3a000640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 0x7f4a240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.743+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4a300cf060 con 0x7f4a3c107e00 2024-03-16T12:19:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.743+0000 7f4a40a00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 0x7f4a240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.744+0000 7f4a40a00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 0x7f4a240a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c02c7b0 tx=0x7f4a2c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.747+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a3009be60 con 0x7f4a3c107e00 2024-03-16T12:19:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.978+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4a3c061c20 con 0x7f4a240a2800 2024-03-16T12:19:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.988+0000 7f4a3a000640 1 -- 172.21.15.98:0/657231039 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4a3c061c20 con 0x7f4a240a2800 2024-03-16T12:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/e3acd339-45d0-42ef-bb67-7eb2c1f94d31 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 msgr2=0x7f4a240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 0x7f4a240a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c02c7b0 tx=0x7f4a2c005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 msgr2=0x7f4a3c19e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e9c0 tx=0x7f4a3000ee80 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 shutdown_connections 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4a240a2800 0x7f4a240a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a3c1099b0 0x7f4a3c1a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a3c108ff0 0x7f4a3c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 --2- 172.21.15.98:0/657231039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a3c107e00 0x7f4a3c19e120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 >> 172.21.15.98:0/657231039 conn(0x7f4a3c076da0 msgr2=0x7f4a3c10c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 shutdown_connections 2024-03-16T12:19:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:08.991+0000 7f4a43e00640 1 -- 172.21.15.98:0/657231039 wait complete. 2024-03-16T12:19:08.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 -- 172.21.15.98:0/2716397182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 msgr2=0x7f4e88109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/2716397182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e88109470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009900 tx=0x7f4e7402bc40 comp rx=0 tx=0).stop 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 -- 172.21.15.98:0/2716397182 shutdown_connections 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/2716397182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 0x7f4e88110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/2716397182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e88109470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/2716397182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 0x7f4e88108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 -- 172.21.15.98:0/2716397182 >> 172.21.15.98:0/2716397182 conn(0x7f4e88076da0 msgr2=0x7f4e880791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.220+0000 7f4e8dc00640 1 -- 172.21.15.98:0/2716397182 shutdown_connections 2024-03-16T12:19:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.221+0000 7f4e8dc00640 1 -- 172.21.15.98:0/2716397182 wait complete. 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.221+0000 7f4e8dc00640 1 Processor -- start 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 -- start start 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 0x7f4e8819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 0x7f4e881a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113ab0 con 0x7f4e88107e00 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113930 con 0x7f4e881099b0 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e8dc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113c30 con 0x7f4e88108ff0 2024-03-16T12:19:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50768/0 (socket says 172.21.15.98:50768) 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 -- 172.21.15.98:0/1343364975 learned_addr learned my addr 172.21.15.98:0/1343364975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e87400640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 0x7f4e881a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86a00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 0x7f4e8819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 -- 172.21.15.98:0/1343364975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 msgr2=0x7f4e881a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 0x7f4e881a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 -- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 msgr2=0x7f4e8819e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.222+0000 7f4e86000640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 0x7f4e8819e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e86000640 1 -- 172.21.15.98:0/1343364975 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e881a5d80 con 0x7f4e88108ff0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e86000640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e7402c8f0 tx=0x7f4e7402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74041070 con 0x7f4e88108ff0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e74004450 con 0x7f4e88108ff0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e881a6010 con 0x7f4e88108ff0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74003d90 con 0x7f4e88108ff0 2024-03-16T12:19:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.223+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e881a6500 con 0x7f4e88108ff0 2024-03-16T12:19:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.224+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8810f0c0 con 0x7f4e88108ff0 2024-03-16T12:19:09.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.227+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4e74044070 con 0x7f4e88108ff0 2024-03-16T12:19:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.228+0000 7f4e7f400640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 0x7f4e6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.228+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4e740f0850 con 0x7f4e88108ff0 2024-03-16T12:19:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.228+0000 7f4e86a00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 0x7f4e6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.229+0000 7f4e86a00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 0x7f4e6c0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e88108e50 tx=0x7f4e78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:09.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.232+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e740bd650 con 0x7f4e88108ff0 2024-03-16T12:19:09.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:08 smithi098 ceph-mon[31633]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:09.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:08 smithi153 ceph-mon[36901]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:09.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:08 smithi173 ceph-mon[36632]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.461+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4e88061c20 con 0x7f4e6c0a2800 2024-03-16T12:19:09.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.488+0000 7f4e7f400640 1 -- 172.21.15.98:0/1343364975 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4e88061c20 con 0x7f4e6c0a2800 2024-03-16T12:19:09.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.490+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 msgr2=0x7f4e6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.490+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 0x7f4e6c0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e88108e50 tx=0x7f4e78005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.490+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 msgr2=0x7f4e8819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.490+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e7402c8f0 tx=0x7f4e7402c660 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 shutdown_connections 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4e6c0a2800 0x7f4e6c0a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e881099b0 0x7f4e881a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e88108ff0 0x7f4e8819e6a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 --2- 172.21.15.98:0/1343364975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e88107e00 0x7f4e8819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 >> 172.21.15.98:0/1343364975 conn(0x7f4e88076da0 msgr2=0x7f4e8810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 shutdown_connections 2024-03-16T12:19:09.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.491+0000 7f4e8dc00640 1 -- 172.21.15.98:0/1343364975 wait complete. 2024-03-16T12:19:09.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T12:19:09.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 -- 172.21.15.98:0/916794316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 msgr2=0x7f52a4103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 --2- 172.21.15.98:0/916794316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4103e40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f529c009ca0 tx=0x7f529c02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 -- 172.21.15.98:0/916794316 shutdown_connections 2024-03-16T12:19:09.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 --2- 172.21.15.98:0/916794316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a41055d0 0x7f52a410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 --2- 172.21.15.98:0/916794316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a4104c10 0x7f52a4105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 --2- 172.21.15.98:0/916794316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4103e40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 -- 172.21.15.98:0/916794316 >> 172.21.15.98:0/916794316 conn(0x7f52a40fef90 msgr2=0x7f52a41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:09.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 -- 172.21.15.98:0/916794316 shutdown_connections 2024-03-16T12:19:09.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.727+0000 7f52a9000640 1 -- 172.21.15.98:0/916794316 wait complete. 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.728+0000 7f52a9000640 1 Processor -- start 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 -- start start 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a4104c10 0x7f52a419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a41055d0 0x7f52a41a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52a410f690 con 0x7f52a4104c10 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52a410f510 con 0x7f52a41055d0 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a9000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52a410f810 con 0x7f52a4103a20 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a4104c10 0x7f52a419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a41055d0 0x7f52a41a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50790/0 (socket says 172.21.15.98:50790) 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.729+0000 7f52a1e00640 1 -- 172.21.15.98:0/1658179037 learned_addr learned my addr 172.21.15.98:0/1658179037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 -- 172.21.15.98:0/1658179037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a41055d0 msgr2=0x7f52a41a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a41055d0 0x7f52a41a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 -- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a4104c10 msgr2=0x7f52a419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a4104c10 0x7f52a419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 -- 172.21.15.98:0/1658179037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a41a19c0 con 0x7f52a4103a20 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a1e00640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f529c000f80 tx=0x7f529c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c041020 con 0x7f52a4103a20 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f529c034cd0 con 0x7f52a4103a20 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a41a1c50 con 0x7f52a4103a20 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c039040 con 0x7f52a4103a20 2024-03-16T12:19:09.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.730+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a41a2160 con 0x7f52a4103a20 2024-03-16T12:19:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.732+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a410ad60 con 0x7f52a4103a20 2024-03-16T12:19:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.733+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f529c044020 con 0x7f52a4103a20 2024-03-16T12:19:09.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.734+0000 7f529aa00640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 0x7f52880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:09.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.735+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f529c0f0a50 con 0x7f52a4103a20 2024-03-16T12:19:09.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.735+0000 7f52a1400640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 0x7f52880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:09.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.735+0000 7f52a1400640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 0x7f52880a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5294009bd0 tx=0x7f5294005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:09.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.738+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f529c04b020 con 0x7f52a4103a20 2024-03-16T12:19:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.938+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f529c0b9820 con 0x7f52a4103a20 2024-03-16T12:19:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.978+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f52a4061cb0 con 0x7f52880a2880 2024-03-16T12:19:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.985+0000 7f529aa00640 1 -- 172.21.15.98:0/1658179037 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f52a4061cb0 con 0x7f52880a2880 2024-03-16T12:19:09.968 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/dedb0f08-2fbc-4337-afa2-6e387c4edd3d 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 msgr2=0x7f52880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 0x7f52880a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5294009bd0 tx=0x7f5294005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 msgr2=0x7f52a4199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f529c000f80 tx=0x7f529c032ec0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 shutdown_connections 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f52880a2880 0x7f52880a4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a41055d0 0x7f52a41a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52a4104c10 0x7f52a419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 --2- 172.21.15.98:0/1658179037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52a4103a20 0x7f52a4199db0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 >> 172.21.15.98:0/1658179037 conn(0x7f52a40fef90 msgr2=0x7f52a4107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 shutdown_connections 2024-03-16T12:19:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:09.988+0000 7f52a9000640 1 -- 172.21.15.98:0/1658179037 wait complete. 2024-03-16T12:19:09.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 -- 172.21.15.98:0/2315058169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc103a00 msgr2=0x7f91dc103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 --2- 172.21.15.98:0/2315058169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc103a00 0x7f91dc103e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009900 tx=0x7f91cc02bc40 comp rx=0 tx=0).stop 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 -- 172.21.15.98:0/2315058169 shutdown_connections 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 --2- 172.21.15.98:0/2315058169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc1055b0 0x7f91dc10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 --2- 172.21.15.98:0/2315058169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 0x7f91dc105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 --2- 172.21.15.98:0/2315058169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc103a00 0x7f91dc103e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 -- 172.21.15.98:0/2315058169 >> 172.21.15.98:0/2315058169 conn(0x7f91dc0fef90 msgr2=0x7f91dc1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:10.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 -- 172.21.15.98:0/2315058169 shutdown_connections 2024-03-16T12:19:10.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.206+0000 7f91e1800640 1 -- 172.21.15.98:0/2315058169 wait complete. 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.207+0000 7f91e1800640 1 Processor -- start 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.207+0000 7f91e1800640 1 -- start start 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 0x7f91dc19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc1055b0 0x7f91dc1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10f6a0 con 0x7f91dc1055b0 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10f520 con 0x7f91dc103a00 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91e1800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10f820 con 0x7f91dc104bf0 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:47866/0 (socket says 172.21.15.98:47866) 2024-03-16T12:19:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 -- 172.21.15.98:0/2762018670 learned_addr learned my addr 172.21.15.98:0/2762018670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91d3e00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 0x7f91dc19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91db400640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc1055b0 0x7f91dc1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 -- 172.21.15.98:0/2762018670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 msgr2=0x7f91dc19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 0x7f91dc19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 -- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc1055b0 msgr2=0x7f91dc1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc1055b0 0x7f91dc1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.208+0000 7f91daa00640 1 -- 172.21.15.98:0/2762018670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91dc1a1930 con 0x7f91dc103a00 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91daa00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f91cc02c8f0 tx=0x7f91cc02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc041070 con 0x7f91dc103a00 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91cc004450 con 0x7f91dc103a00 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc034440 con 0x7f91dc103a00 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91dc1a1bc0 con 0x7f91dc103a00 2024-03-16T12:19:10.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.209+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91dc1a2100 con 0x7f91dc103a00 2024-03-16T12:19:10.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.210+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91dc10acc0 con 0x7f91dc103a00 2024-03-16T12:19:10.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.212+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f91cc044070 con 0x7f91dc103a00 2024-03-16T12:19:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.213+0000 7f91d3400640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 0x7f91c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.213+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f91cc0f0ca0 con 0x7f91dc103a00 2024-03-16T12:19:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.214+0000 7f91d3e00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 0x7f91c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.214+0000 7f91d3e00640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 0x7f91c00a4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009bd0 tx=0x7f91c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:10.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.216+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91cc0bda50 con 0x7f91dc103a00 2024-03-16T12:19:10.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:09 smithi098 ceph-mon[31633]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:09 smithi098 ceph-mon[31633]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-03-16T12:19:10.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:09 smithi098 ceph-mon[31633]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.254 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:19:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: 2024-03-16T12:19:09.920+0000 7fb9af21a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T12:19:10.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:09 smithi153 ceph-mon[36901]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:09 smithi153 ceph-mon[36901]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-03-16T12:19:10.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:09 smithi153 ceph-mon[36901]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:09 smithi173 ceph-mon[36632]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:09 smithi173 ceph-mon[36632]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 1 op/s 2024-03-16T12:19:10.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:09 smithi173 ceph-mon[36632]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.454+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91dc02d400 con 0x7f91dc103a00 2024-03-16T12:19:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.455+0000 7f91d3400640 1 -- 172.21.15.98:0/2762018670 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7f91cc0bd350 con 0x7f91dc103a00 2024-03-16T12:19:10.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:10.438 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:19:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:19:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 msgr2=0x7f91c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 0x7f91c00a4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009bd0 tx=0x7f91c4005f90 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 msgr2=0x7f91dc199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f91cc02c8f0 tx=0x7f91cc02c660 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 shutdown_connections 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91c00a2910 0x7f91c00a4dd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91dc1055b0 0x7f91dc1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91dc104bf0 0x7f91dc19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 --2- 172.21.15.98:0/2762018670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91dc103a00 0x7f91dc199d20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 >> 172.21.15.98:0/2762018670 conn(0x7f91dc0fef90 msgr2=0x7f91dc107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 shutdown_connections 2024-03-16T12:19:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.459+0000 7f91e1800640 1 -- 172.21.15.98:0/2762018670 wait complete. 2024-03-16T12:19:10.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 -- 172.21.15.98:0/677984210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 msgr2=0x7fcf18108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 --2- 172.21.15.98:0/677984210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf18108220 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009900 tx=0x7fcf0c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 -- 172.21.15.98:0/677984210 shutdown_connections 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 --2- 172.21.15.98:0/677984210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 0x7fcf18110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 --2- 172.21.15.98:0/677984210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf18109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 --2- 172.21.15.98:0/677984210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf18108220 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 -- 172.21.15.98:0/677984210 >> 172.21.15.98:0/677984210 conn(0x7fcf18076dc0 msgr2=0x7fcf180791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 -- 172.21.15.98:0/677984210 shutdown_connections 2024-03-16T12:19:10.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.680+0000 7fcf20400640 1 -- 172.21.15.98:0/677984210 wait complete. 2024-03-16T12:19:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.681+0000 7fcf20400640 1 Processor -- start 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.681+0000 7fcf20400640 1 -- start start 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf1819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 0x7fcf181a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18113a50 con 0x7fcf181099b0 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181138d0 con 0x7fcf18107e00 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf20400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18113bd0 con 0x7fcf18108ff0 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1da00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf1819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50822/0 (socket says 172.21.15.98:50822) 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 -- 172.21.15.98:0/1994978869 learned_addr learned my addr 172.21.15.98:0/1994978869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1e400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 0x7fcf181a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 -- 172.21.15.98:0/1994978869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 msgr2=0x7fcf1819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf1819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 -- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 msgr2=0x7fcf181a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 0x7fcf181a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1d000640 1 -- 172.21.15.98:0/1994978869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf181a5d20 con 0x7fcf18108ff0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.682+0000 7fcf1da00640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf1819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf1d000640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400ea60 tx=0x7fcf0400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400cde0 con 0x7fcf18108ff0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf040035f0 con 0x7fcf18108ff0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04010670 con 0x7fcf18108ff0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181a5fb0 con 0x7fcf18108ff0 2024-03-16T12:19:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.683+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf181a6520 con 0x7fcf18108ff0 2024-03-16T12:19:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.684+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1810f120 con 0x7fcf18108ff0 2024-03-16T12:19:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.686+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcf040037a0 con 0x7fcf18108ff0 2024-03-16T12:19:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.687+0000 7fcf16000640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 0x7fcf000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.688+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcf040cf100 con 0x7fcf18108ff0 2024-03-16T12:19:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.688+0000 7fcf1da00640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 0x7fcf000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.688+0000 7fcf1da00640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 0x7fcf000a4dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.691+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf0409bda0 con 0x7fcf18108ff0 2024-03-16T12:19:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.891+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf1802d400 con 0x7fcf18108ff0 2024-03-16T12:19:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.893+0000 7fcf16000640 1 -- 172.21.15.98:0/1994978869 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fcf0409b6a0 con 0x7fcf18108ff0 2024-03-16T12:19:10.875 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:10.876 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.895+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 msgr2=0x7fcf000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.895+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 0x7fcf000a4dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.895+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 msgr2=0x7fcf1819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.895+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400ea60 tx=0x7fcf0400ef20 comp rx=0 tx=0).stop 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 shutdown_connections 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf000a2910 0x7fcf000a4dd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf181099b0 0x7fcf181a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf18108ff0 0x7fcf1819e640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 --2- 172.21.15.98:0/1994978869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf18107e00 0x7fcf1819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 >> 172.21.15.98:0/1994978869 conn(0x7fcf18076dc0 msgr2=0x7fcf1810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 shutdown_connections 2024-03-16T12:19:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:10.896+0000 7fcf20400640 1 -- 172.21.15.98:0/1994978869 wait complete. 2024-03-16T12:19:10.897 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T12:19:10.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T12:19:11.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.112+0000 7fe42b400640 1 -- 172.21.15.98:0/980512334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240761a0 msgr2=0x7fe4240765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.112+0000 7fe42b400640 1 --2- 172.21.15.98:0/980512334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240761a0 0x7fe4240765c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe418009900 tx=0x7fe41802bc30 comp rx=0 tx=0).stop 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 -- 172.21.15.98:0/980512334 shutdown_connections 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 --2- 172.21.15.98:0/980512334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe424078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 --2- 172.21.15.98:0/980512334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240773e0 0x7fe424077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 --2- 172.21.15.98:0/980512334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240761a0 0x7fe4240765c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 -- 172.21.15.98:0/980512334 >> 172.21.15.98:0/980512334 conn(0x7fe4240fef90 msgr2=0x7fe4241013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 -- 172.21.15.98:0/980512334 shutdown_connections 2024-03-16T12:19:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.113+0000 7fe42b400640 1 -- 172.21.15.98:0/980512334 wait complete. 2024-03-16T12:19:11.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 Processor -- start 2024-03-16T12:19:11.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 -- start start 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 0x7fe42419e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe4241a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe424113a50 con 0x7fe4240761a0 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241138d0 con 0x7fe4240773e0 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.114+0000 7fe42b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe424113bd0 con 0x7fe424077d80 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe428a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 0x7fe42419e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:47908/0 (socket says 172.21.15.98:47908) 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe428a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 0x7fe42419e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47162/0 (socket says 172.21.15.98:47162) 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 -- 172.21.15.98:0/1630019070 learned_addr learned my addr 172.21.15.98:0/1630019070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe429400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe4241a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 -- 172.21.15.98:0/1630019070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 msgr2=0x7fe4241a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe4241a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 -- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 msgr2=0x7fe42419e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 0x7fe42419e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 -- 172.21.15.98:0/1630019070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241a5d40 con 0x7fe4240773e0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe429400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe4241a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe423e00640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe41000e9f0 tx=0x7fe41000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.115+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41000cd80 con 0x7fe4240773e0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.116+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe410010910 con 0x7fe4240773e0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.116+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241a5fd0 con 0x7fe4240773e0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.116+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe410018990 con 0x7fe4240773e0 2024-03-16T12:19:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.116+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a6540 con 0x7fe4240773e0 2024-03-16T12:19:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.117+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe424110500 con 0x7fe4240773e0 2024-03-16T12:19:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.119+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe410010ac0 con 0x7fe4240773e0 2024-03-16T12:19:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.120+0000 7fe421600640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 0x7fe40c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.120+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe4100cf110 con 0x7fe4240773e0 2024-03-16T12:19:11.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.121+0000 7fe428a00640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 0x7fe40c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.122+0000 7fe428a00640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 0x7fe40c0a4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe41802c7b0 tx=0x7fe418005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.124+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe41009b760 con 0x7fe4240773e0 2024-03-16T12:19:11.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:10 smithi098 ceph-mon[31633]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:10 smithi098 ceph-mon[31633]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-16T12:19:11.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:10 smithi098 ceph-mon[31633]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:10 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2762018670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:10 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1994978869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:10 smithi153 ceph-mon[36901]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:10 smithi153 ceph-mon[36901]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-16T12:19:11.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:10 smithi153 ceph-mon[36901]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:10 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2762018670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:10 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1994978869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.318+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe424061c20 con 0x7fe40c0a2910 2024-03-16T12:19:11.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:10 smithi173 ceph-mon[36632]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:10 smithi173 ceph-mon[36632]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-16T12:19:11.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:10 smithi173 ceph-mon[36632]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:19:11.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:10 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2762018670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:10 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1994978869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.339+0000 7fe421600640 1 -- 172.21.15.98:0/1630019070 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340362 (secure 0 0 0) 0x7fe424061c20 con 0x7fe40c0a2910 2024-03-16T12:19:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:11.329 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:19:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 msgr2=0x7fe40c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 0x7fe40c0a4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe41802c7b0 tx=0x7fe418005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 msgr2=0x7fe42419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe41000e9f0 tx=0x7fe41000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 shutdown_connections 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe40c0a2910 0x7fe40c0a4dd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424077d80 0x7fe4241a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4240773e0 0x7fe42419e660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 --2- 172.21.15.98:0/1630019070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4240761a0 0x7fe42419e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 >> 172.21.15.98:0/1630019070 conn(0x7fe4240fef90 msgr2=0x7fe4241006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 shutdown_connections 2024-03-16T12:19:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.346+0000 7fe42b400640 1 -- 172.21.15.98:0/1630019070 wait complete. 2024-03-16T12:19:11.348 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-16T12:19:10.780377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597355,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":148,"num_read_kb":127,"num_write":303,"num_write_kb":2984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383424,"kb_used_data":13460,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558400,"statfs":{"total":1151940427776,"available":1151547801600,"internally_reserved":0,"allocated":13783040,"data_stored":7533008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4128,"num_objects":33,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":76,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005417"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859526+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:18:46.859526+0000","last_peered":"2024-03-16T12:18:46.859526+0000","last_clean":"2024-03-16T12:18:46.859526+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:18:46.859526+0000","last_undegraded":"2024-03-16T12:18:46.859526+0000","last_fullsized":"2024-03-16T12:18:46.859526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557727+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:18:45.557727+0000","last_peered":"2024-03-16T12:18:45.557727+0000","last_clean":"2024-03-16T12:18:45.557727+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:18:45.557727+0000","last_undegraded":"2024-03-16T12:18:45.557727+0000","last_fullsized":"2024-03-16T12:18:45.557727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557290+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:18:45.557290+0000","last_peered":"2024-03-16T12:18:45.557290+0000","last_clean":"2024-03-16T12:18:45.557290+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:18:45.557290+0000","last_undegraded":"2024-03-16T12:18:45.557290+0000","last_fullsized":"2024-03-16T12:18:45.557290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859435+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:18:46.859435+0000","last_peered":"2024-03-16T12:18:46.859435+0000","last_clean":"2024-03-16T12:18:46.859435+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:18:46.859435+0000","last_undegraded":"2024-03-16T12:18:46.859435+0000","last_fullsized":"2024-03-16T12:18:46.859435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859032+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:18:46.859032+0000","last_peered":"2024-03-16T12:18:46.859032+0000","last_clean":"2024-03-16T12:18:46.859032+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:18:46.859032+0000","last_undegraded":"2024-03-16T12:18:46.859032+0000","last_fullsized":"2024-03-16T12:18:46.859032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858382+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:18:46.858382+0000","last_peered":"2024-03-16T12:18:46.858382+0000","last_clean":"2024-03-16T12:18:46.858382+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:18:46.858382+0000","last_undegraded":"2024-03-16T12:18:46.858382+0000","last_fullsized":"2024-03-16T12:18:46.858382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556016+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:18:45.556016+0000","last_peered":"2024-03-16T12:18:45.556016+0000","last_clean":"2024-03-16T12:18:45.556016+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:18:45.556016+0000","last_undegraded":"2024-03-16T12:18:45.556016+0000","last_fullsized":"2024-03-16T12:18:45.556016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.351902+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:18:46.351902+0000","last_peered":"2024-03-16T12:18:46.351902+0000","last_clean":"2024-03-16T12:18:46.351902+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:18:46.351902+0000","last_undegraded":"2024-03-16T12:18:46.351902+0000","last_fullsized":"2024-03-16T12:18:46.351902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859210+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:18:46.859210+0000","last_peered":"2024-03-16T12:18:46.859210+0000","last_clean":"2024-03-16T12:18:46.859210+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:18:46.859210+0000","last_undegraded":"2024-03-16T12:18:46.859210+0000","last_fullsized":"2024-03-16T12:18:46.859210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858497+0000","last_change":"2024-03-16T12:18:40.503141+0000","last_active":"2024-03-16T12:18:46.858497+0000","last_peered":"2024-03-16T12:18:46.858497+0000","last_clean":"2024-03-16T12:18:46.858497+0000","last_became_active":"2024-03-16T12:18:40.502789+0000","last_became_peered":"2024-03-16T12:18:40.502789+0000","last_unstale":"2024-03-16T12:18:46.858497+0000","last_undegraded":"2024-03-16T12:18:46.858497+0000","last_fullsized":"2024-03-16T12:18:46.858497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:37.220085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556575+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:18:45.556575+0000","last_peered":"2024-03-16T12:18:45.556575+0000","last_clean":"2024-03-16T12:18:45.556575+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:18:45.556575+0000","last_undegraded":"2024-03-16T12:18:45.556575+0000","last_fullsized":"2024-03-16T12:18:45.556575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859352+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:18:46.859352+0000","last_peered":"2024-03-16T12:18:46.859352+0000","last_clean":"2024-03-16T12:18:46.859352+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:18:46.859352+0000","last_undegraded":"2024-03-16T12:18:46.859352+0000","last_fullsized":"2024-03-16T12:18:46.859352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559046+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:18:45.559046+0000","last_peered":"2024-03-16T12:18:45.559046+0000","last_clean":"2024-03-16T12:18:45.559046+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:18:45.559046+0000","last_undegraded":"2024-03-16T12:18:45.559046+0000","last_fullsized":"2024-03-16T12:18:45.559046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556137+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:18:45.556137+0000","last_peered":"2024-03-16T12:18:45.556137+0000","last_clean":"2024-03-16T12:18:45.556137+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:18:45.556137+0000","last_undegraded":"2024-03-16T12:18:45.556137+0000","last_fullsized":"2024-03-16T12:18:45.556137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556398+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:18:45.556398+0000","last_peered":"2024-03-16T12:18:45.556398+0000","last_clean":"2024-03-16T12:18:45.556398+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:18:45.556398+0000","last_undegraded":"2024-03-16T12:18:45.556398+0000","last_fullsized":"2024-03-16T12:18:45.556398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858431+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:18:46.858431+0000","last_peered":"2024-03-16T12:18:46.858431+0000","last_clean":"2024-03-16T12:18:46.858431+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:18:46.858431+0000","last_undegraded":"2024-03-16T12:18:46.858431+0000","last_fullsized":"2024-03-16T12:18:46.858431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859429+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:18:46.859429+0000","last_peered":"2024-03-16T12:18:46.859429+0000","last_clean":"2024-03-16T12:18:46.859429+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:18:46.859429+0000","last_undegraded":"2024-03-16T12:18:46.859429+0000","last_fullsized":"2024-03-16T12:18:46.859429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.861061+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:18:46.861061+0000","last_peered":"2024-03-16T12:18:46.861061+0000","last_clean":"2024-03-16T12:18:46.861061+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:18:46.861061+0000","last_undegraded":"2024-03-16T12:18:46.861061+0000","last_fullsized":"2024-03-16T12:18:46.861061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858863+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:18:46.858863+0000","last_peered":"2024-03-16T12:18:46.858863+0000","last_clean":"2024-03-16T12:18:46.858863+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:18:46.858863+0000","last_undegraded":"2024-03-16T12:18:46.858863+0000","last_fullsized":"2024-03-16T12:18:46.858863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859569+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:18:46.859569+0000","last_peered":"2024-03-16T12:18:46.859569+0000","last_clean":"2024-03-16T12:18:46.859569+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:18:46.859569+0000","last_undegraded":"2024-03-16T12:18:46.859569+0000","last_fullsized":"2024-03-16T12:18:46.859569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.932837+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:06.932837+0000","last_peered":"2024-03-16T12:19:06.932837+0000","last_clean":"2024-03-16T12:19:06.932837+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:06.932837+0000","last_undegraded":"2024-03-16T12:19:06.932837+0000","last_fullsized":"2024-03-16T12:19:06.932837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352729+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:18:46.352729+0000","last_peered":"2024-03-16T12:18:46.352729+0000","last_clean":"2024-03-16T12:18:46.352729+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:18:46.352729+0000","last_undegraded":"2024-03-16T12:18:46.352729+0000","last_fullsized":"2024-03-16T12:18:46.352729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.857987+0000","last_change":"2024-03-16T12:18:44.590453+0000","last_active":"2024-03-16T12:18:46.857987+0000","last_peered":"2024-03-16T12:18:46.857987+0000","last_clean":"2024-03-16T12:18:46.857987+0000","last_became_active":"2024-03-16T12:18:44.589804+0000","last_became_peered":"2024-03-16T12:18:44.589804+0000","last_unstale":"2024-03-16T12:18:46.857987+0000","last_undegraded":"2024-03-16T12:18:46.857987+0000","last_fullsized":"2024-03-16T12:18:46.857987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:05.144303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353104+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:18:46.353104+0000","last_peered":"2024-03-16T12:18:46.353104+0000","last_clean":"2024-03-16T12:18:46.353104+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:18:46.353104+0000","last_undegraded":"2024-03-16T12:18:46.353104+0000","last_fullsized":"2024-03-16T12:18:46.353104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557830+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:18:45.557830+0000","last_peered":"2024-03-16T12:18:45.557830+0000","last_clean":"2024-03-16T12:18:45.557830+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:18:45.557830+0000","last_undegraded":"2024-03-16T12:18:45.557830+0000","last_fullsized":"2024-03-16T12:18:45.557830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859516+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:18:46.859516+0000","last_peered":"2024-03-16T12:18:46.859516+0000","last_clean":"2024-03-16T12:18:46.859516+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:18:46.859516+0000","last_undegraded":"2024-03-16T12:18:46.859516+0000","last_fullsized":"2024-03-16T12:18:46.859516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.927222+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:06.927222+0000","last_peered":"2024-03-16T12:19:06.927222+0000","last_clean":"2024-03-16T12:19:06.927222+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:06.927222+0000","last_undegraded":"2024-03-16T12:19:06.927222+0000","last_fullsized":"2024-03-16T12:19:06.927222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858760+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:18:46.858760+0000","last_peered":"2024-03-16T12:18:46.858760+0000","last_clean":"2024-03-16T12:18:46.858760+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:18:46.858760+0000","last_undegraded":"2024-03-16T12:18:46.858760+0000","last_fullsized":"2024-03-16T12:18:46.858760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858351+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:18:46.858351+0000","last_peered":"2024-03-16T12:18:46.858351+0000","last_clean":"2024-03-16T12:18:46.858351+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:18:46.858351+0000","last_undegraded":"2024-03-16T12:18:46.858351+0000","last_fullsized":"2024-03-16T12:18:46.858351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859795+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:18:46.859795+0000","last_peered":"2024-03-16T12:18:46.859795+0000","last_clean":"2024-03-16T12:18:46.859795+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:18:46.859795+0000","last_undegraded":"2024-03-16T12:18:46.859795+0000","last_fullsized":"2024-03-16T12:18:46.859795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219215+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:18:46.219215+0000","last_peered":"2024-03-16T12:18:46.219215+0000","last_clean":"2024-03-16T12:18:46.219215+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:18:46.219215+0000","last_undegraded":"2024-03-16T12:18:46.219215+0000","last_fullsized":"2024-03-16T12:18:46.219215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858856+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:18:46.858856+0000","last_peered":"2024-03-16T12:18:46.858856+0000","last_clean":"2024-03-16T12:18:46.858856+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:18:46.858856+0000","last_undegraded":"2024-03-16T12:18:46.858856+0000","last_fullsized":"2024-03-16T12:18:46.858856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.489025+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:19:07.489025+0000","last_peered":"2024-03-16T12:19:07.489025+0000","last_clean":"2024-03-16T12:19:07.489025+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:19:07.489025+0000","last_undegraded":"2024-03-16T12:19:07.489025+0000","last_fullsized":"2024-03-16T12:19:07.489025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858739+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:18:46.858739+0000","last_peered":"2024-03-16T12:18:46.858739+0000","last_clean":"2024-03-16T12:18:46.858739+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:18:46.858739+0000","last_undegraded":"2024-03-16T12:18:46.858739+0000","last_fullsized":"2024-03-16T12:18:46.858739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859150+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:18:46.859150+0000","last_peered":"2024-03-16T12:18:46.859150+0000","last_clean":"2024-03-16T12:18:46.859150+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:18:46.859150+0000","last_undegraded":"2024-03-16T12:18:46.859150+0000","last_fullsized":"2024-03-16T12:18:46.859150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858356+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:18:46.858356+0000","last_peered":"2024-03-16T12:18:46.858356+0000","last_clean":"2024-03-16T12:18:46.858356+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:18:46.858356+0000","last_undegraded":"2024-03-16T12:18:46.858356+0000","last_fullsized":"2024-03-16T12:18:46.858356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219265+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:18:46.219265+0000","last_peered":"2024-03-16T12:18:46.219265+0000","last_clean":"2024-03-16T12:18:46.219265+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:18:46.219265+0000","last_undegraded":"2024-03-16T12:18:46.219265+0000","last_fullsized":"2024-03-16T12:18:46.219265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859712+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:18:46.859712+0000","last_peered":"2024-03-16T12:18:46.859712+0000","last_clean":"2024-03-16T12:18:46.859712+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:18:46.859712+0000","last_undegraded":"2024-03-16T12:18:46.859712+0000","last_fullsized":"2024-03-16T12:18:46.859712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.933723+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:06.933723+0000","last_peered":"2024-03-16T12:19:06.933723+0000","last_clean":"2024-03-16T12:19:06.933723+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:06.933723+0000","last_undegraded":"2024-03-16T12:19:06.933723+0000","last_fullsized":"2024-03-16T12:19:06.933723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352163+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:18:46.352163+0000","last_peered":"2024-03-16T12:18:46.352163+0000","last_clean":"2024-03-16T12:18:46.352163+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:18:46.352163+0000","last_undegraded":"2024-03-16T12:18:46.352163+0000","last_fullsized":"2024-03-16T12:18:46.352163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557872+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:18:45.557872+0000","last_peered":"2024-03-16T12:18:45.557872+0000","last_clean":"2024-03-16T12:18:45.557872+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:18:45.557872+0000","last_undegraded":"2024-03-16T12:18:45.557872+0000","last_fullsized":"2024-03-16T12:18:45.557872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218865+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:18:46.218865+0000","last_peered":"2024-03-16T12:18:46.218865+0000","last_clean":"2024-03-16T12:18:46.218865+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:18:46.218865+0000","last_undegraded":"2024-03-16T12:18:46.218865+0000","last_fullsized":"2024-03-16T12:18:46.218865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859609+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:18:46.859609+0000","last_peered":"2024-03-16T12:18:46.859609+0000","last_clean":"2024-03-16T12:18:46.859609+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:18:46.859609+0000","last_undegraded":"2024-03-16T12:18:46.859609+0000","last_fullsized":"2024-03-16T12:18:46.859609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352626+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:18:46.352626+0000","last_peered":"2024-03-16T12:18:46.352626+0000","last_clean":"2024-03-16T12:18:46.352626+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:18:46.352626+0000","last_undegraded":"2024-03-16T12:18:46.352626+0000","last_fullsized":"2024-03-16T12:18:46.352626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907419+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:19:05.907419+0000","last_peered":"2024-03-16T12:19:05.907419+0000","last_clean":"2024-03-16T12:19:05.907419+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:19:05.907419+0000","last_undegraded":"2024-03-16T12:19:05.907419+0000","last_fullsized":"2024-03-16T12:19:05.907419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858813+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:18:46.858813+0000","last_peered":"2024-03-16T12:18:46.858813+0000","last_clean":"2024-03-16T12:18:46.858813+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:18:46.858813+0000","last_undegraded":"2024-03-16T12:18:46.858813+0000","last_fullsized":"2024-03-16T12:18:46.858813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859796+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:18:46.859796+0000","last_peered":"2024-03-16T12:18:46.859796+0000","last_clean":"2024-03-16T12:18:46.859796+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:18:46.859796+0000","last_undegraded":"2024-03-16T12:18:46.859796+0000","last_fullsized":"2024-03-16T12:18:46.859796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.906433+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:05.906433+0000","last_peered":"2024-03-16T12:19:05.906433+0000","last_clean":"2024-03-16T12:19:05.906433+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:05.906433+0000","last_undegraded":"2024-03-16T12:19:05.906433+0000","last_fullsized":"2024-03-16T12:19:05.906433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557834+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:18:45.557834+0000","last_peered":"2024-03-16T12:18:45.557834+0000","last_clean":"2024-03-16T12:18:45.557834+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:18:45.557834+0000","last_undegraded":"2024-03-16T12:18:45.557834+0000","last_fullsized":"2024-03-16T12:18:45.557834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859678+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:18:46.859678+0000","last_peered":"2024-03-16T12:18:46.859678+0000","last_clean":"2024-03-16T12:18:46.859678+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:18:46.859678+0000","last_undegraded":"2024-03-16T12:18:46.859678+0000","last_fullsized":"2024-03-16T12:18:46.859678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907938+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:05.907938+0000","last_peered":"2024-03-16T12:19:05.907938+0000","last_clean":"2024-03-16T12:19:05.907938+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:05.907938+0000","last_undegraded":"2024-03-16T12:19:05.907938+0000","last_fullsized":"2024-03-16T12:19:05.907938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858941+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:18:46.858941+0000","last_peered":"2024-03-16T12:18:46.858941+0000","last_clean":"2024-03-16T12:18:46.858941+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:18:46.858941+0000","last_undegraded":"2024-03-16T12:18:46.858941+0000","last_fullsized":"2024-03-16T12:18:46.858941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859732+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:18:46.859732+0000","last_peered":"2024-03-16T12:18:46.859732+0000","last_clean":"2024-03-16T12:18:46.859732+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:18:46.859732+0000","last_undegraded":"2024-03-16T12:18:46.859732+0000","last_fullsized":"2024-03-16T12:18:46.859732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.905660+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:19:05.905660+0000","last_peered":"2024-03-16T12:19:05.905660+0000","last_clean":"2024-03-16T12:19:05.905660+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:19:05.905660+0000","last_undegraded":"2024-03-16T12:19:05.905660+0000","last_fullsized":"2024-03-16T12:19:05.905660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858919+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:18:46.858919+0000","last_peered":"2024-03-16T12:18:46.858919+0000","last_clean":"2024-03-16T12:18:46.858919+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:18:46.858919+0000","last_undegraded":"2024-03-16T12:18:46.858919+0000","last_fullsized":"2024-03-16T12:18:46.858919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219498+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:18:46.219498+0000","last_peered":"2024-03-16T12:18:46.219498+0000","last_clean":"2024-03-16T12:18:46.219498+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:18:46.219498+0000","last_undegraded":"2024-03-16T12:18:46.219498+0000","last_fullsized":"2024-03-16T12:18:46.219498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859382+0000","last_change":"2024-03-16T12:18:39.988134+0000","last_active":"2024-03-16T12:18:46.859382+0000","last_peered":"2024-03-16T12:18:46.859382+0000","last_clean":"2024-03-16T12:18:46.859382+0000","last_became_active":"2024-03-16T12:18:39.987772+0000","last_became_peered":"2024-03-16T12:18:39.987772+0000","last_unstale":"2024-03-16T12:18:46.859382+0000","last_undegraded":"2024-03-16T12:18:46.859382+0000","last_fullsized":"2024-03-16T12:18:46.859382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:49.889547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559808+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:18:45.559808+0000","last_peered":"2024-03-16T12:18:45.559808+0000","last_clean":"2024-03-16T12:18:45.559808+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:18:45.559808+0000","last_undegraded":"2024-03-16T12:18:45.559808+0000","last_fullsized":"2024-03-16T12:18:45.559808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859114+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:18:46.859114+0000","last_peered":"2024-03-16T12:18:46.859114+0000","last_clean":"2024-03-16T12:18:46.859114+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:18:46.859114+0000","last_undegraded":"2024-03-16T12:18:46.859114+0000","last_fullsized":"2024-03-16T12:18:46.859114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907823+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:05.907823+0000","last_peered":"2024-03-16T12:19:05.907823+0000","last_clean":"2024-03-16T12:19:05.907823+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:05.907823+0000","last_undegraded":"2024-03-16T12:19:05.907823+0000","last_fullsized":"2024-03-16T12:19:05.907823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556262+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:18:45.556262+0000","last_peered":"2024-03-16T12:18:45.556262+0000","last_clean":"2024-03-16T12:18:45.556262+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:18:45.556262+0000","last_undegraded":"2024-03-16T12:18:45.556262+0000","last_fullsized":"2024-03-16T12:18:45.556262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.860624+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:18:46.860624+0000","last_peered":"2024-03-16T12:18:46.860624+0000","last_clean":"2024-03-16T12:18:46.860624+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:18:46.860624+0000","last_undegraded":"2024-03-16T12:18:46.860624+0000","last_fullsized":"2024-03-16T12:18:46.860624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.490502+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:19:07.490502+0000","last_peered":"2024-03-16T12:19:07.490502+0000","last_clean":"2024-03-16T12:19:07.490502+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:19:07.490502+0000","last_undegraded":"2024-03-16T12:19:07.490502+0000","last_fullsized":"2024-03-16T12:19:07.490502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556058+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:18:45.556058+0000","last_peered":"2024-03-16T12:18:45.556058+0000","last_clean":"2024-03-16T12:18:45.556058+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:18:45.556058+0000","last_undegraded":"2024-03-16T12:18:45.556058+0000","last_fullsized":"2024-03-16T12:18:45.556058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561332+0000","last_change":"2024-03-16T12:18:44.589442+0000","last_active":"2024-03-16T12:18:45.561332+0000","last_peered":"2024-03-16T12:18:45.561332+0000","last_clean":"2024-03-16T12:18:45.561332+0000","last_became_active":"2024-03-16T12:18:44.588933+0000","last_became_peered":"2024-03-16T12:18:44.588933+0000","last_unstale":"2024-03-16T12:18:45.561332+0000","last_undegraded":"2024-03-16T12:18:45.561332+0000","last_fullsized":"2024-03-16T12:18:45.561332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:07:10.860786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.934282+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:06.934282+0000","last_peered":"2024-03-16T12:19:06.934282+0000","last_clean":"2024-03-16T12:19:06.934282+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:06.934282+0000","last_undegraded":"2024-03-16T12:19:06.934282+0000","last_fullsized":"2024-03-16T12:19:06.934282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218791+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:18:46.218791+0000","last_peered":"2024-03-16T12:18:46.218791+0000","last_clean":"2024-03-16T12:18:46.218791+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:18:46.218791+0000","last_undegraded":"2024-03-16T12:18:46.218791+0000","last_fullsized":"2024-03-16T12:18:46.218791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218822+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:18:46.218822+0000","last_peered":"2024-03-16T12:18:46.218822+0000","last_clean":"2024-03-16T12:18:46.218822+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:18:46.218822+0000","last_undegraded":"2024-03-16T12:18:46.218822+0000","last_fullsized":"2024-03-16T12:18:46.218822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.486344+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:07.486344+0000","last_peered":"2024-03-16T12:19:07.486344+0000","last_clean":"2024-03-16T12:19:07.486344+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:07.486344+0000","last_undegraded":"2024-03-16T12:19:07.486344+0000","last_fullsized":"2024-03-16T12:19:07.486344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858850+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:18:46.858850+0000","last_peered":"2024-03-16T12:18:46.858850+0000","last_clean":"2024-03-16T12:18:46.858850+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:18:46.858850+0000","last_undegraded":"2024-03-16T12:18:46.858850+0000","last_fullsized":"2024-03-16T12:18:46.858850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858623+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:18:46.858623+0000","last_peered":"2024-03-16T12:18:46.858623+0000","last_clean":"2024-03-16T12:18:46.858623+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:18:46.858623+0000","last_undegraded":"2024-03-16T12:18:46.858623+0000","last_fullsized":"2024-03-16T12:18:46.858623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.860566+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:18:46.860566+0000","last_peered":"2024-03-16T12:18:46.860566+0000","last_clean":"2024-03-16T12:18:46.860566+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:18:46.860566+0000","last_undegraded":"2024-03-16T12:18:46.860566+0000","last_fullsized":"2024-03-16T12:18:46.860566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858658+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:18:46.858658+0000","last_peered":"2024-03-16T12:18:46.858658+0000","last_clean":"2024-03-16T12:18:46.858658+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:18:46.858658+0000","last_undegraded":"2024-03-16T12:18:46.858658+0000","last_fullsized":"2024-03-16T12:18:46.858658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559274+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:18:45.559274+0000","last_peered":"2024-03-16T12:18:45.559274+0000","last_clean":"2024-03-16T12:18:45.559274+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:18:45.559274+0000","last_undegraded":"2024-03-16T12:18:45.559274+0000","last_fullsized":"2024-03-16T12:18:45.559274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556218+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:18:45.556218+0000","last_peered":"2024-03-16T12:18:45.556218+0000","last_clean":"2024-03-16T12:18:45.556218+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:18:45.556218+0000","last_undegraded":"2024-03-16T12:18:45.556218+0000","last_fullsized":"2024-03-16T12:18:45.556218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858105+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:18:46.858105+0000","last_peered":"2024-03-16T12:18:46.858105+0000","last_clean":"2024-03-16T12:18:46.858105+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:18:46.858105+0000","last_undegraded":"2024-03-16T12:18:46.858105+0000","last_fullsized":"2024-03-16T12:18:46.858105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561198+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:18:45.561198+0000","last_peered":"2024-03-16T12:18:45.561198+0000","last_clean":"2024-03-16T12:18:45.561198+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:18:45.561198+0000","last_undegraded":"2024-03-16T12:18:45.561198+0000","last_fullsized":"2024-03-16T12:18:45.561198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218929+0000","last_change":"2024-03-16T12:18:40.505877+0000","last_active":"2024-03-16T12:18:46.218929+0000","last_peered":"2024-03-16T12:18:46.218929+0000","last_clean":"2024-03-16T12:18:46.218929+0000","last_became_active":"2024-03-16T12:18:40.504655+0000","last_became_peered":"2024-03-16T12:18:40.504655+0000","last_unstale":"2024-03-16T12:18:46.218929+0000","last_undegraded":"2024-03-16T12:18:46.218929+0000","last_fullsized":"2024-03-16T12:18:46.218929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:27.043333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.488421+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:19:07.488421+0000","last_peered":"2024-03-16T12:19:07.488421+0000","last_clean":"2024-03-16T12:19:07.488421+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:19:07.488421+0000","last_undegraded":"2024-03-16T12:19:07.488421+0000","last_fullsized":"2024-03-16T12:19:07.488421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556553+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:18:45.556553+0000","last_peered":"2024-03-16T12:18:45.556553+0000","last_clean":"2024-03-16T12:18:45.556553+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:18:45.556553+0000","last_undegraded":"2024-03-16T12:18:45.556553+0000","last_fullsized":"2024-03-16T12:18:45.556553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352435+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:18:46.352435+0000","last_peered":"2024-03-16T12:18:46.352435+0000","last_clean":"2024-03-16T12:18:46.352435+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:18:46.352435+0000","last_undegraded":"2024-03-16T12:18:46.352435+0000","last_fullsized":"2024-03-16T12:18:46.352435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.485920+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:07.485920+0000","last_peered":"2024-03-16T12:19:07.485920+0000","last_clean":"2024-03-16T12:19:07.485920+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:07.485920+0000","last_undegraded":"2024-03-16T12:19:07.485920+0000","last_fullsized":"2024-03-16T12:19:07.485920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.555947+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:18:45.555947+0000","last_peered":"2024-03-16T12:18:45.555947+0000","last_clean":"2024-03-16T12:18:45.555947+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:18:45.555947+0000","last_undegraded":"2024-03-16T12:18:45.555947+0000","last_fullsized":"2024-03-16T12:18:45.555947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219248+0000","last_change":"2024-03-16T12:18:44.588798+0000","last_active":"2024-03-16T12:18:46.219248+0000","last_peered":"2024-03-16T12:18:46.219248+0000","last_clean":"2024-03-16T12:18:46.219248+0000","last_became_active":"2024-03-16T12:18:44.588498+0000","last_became_peered":"2024-03-16T12:18:44.588498+0000","last_unstale":"2024-03-16T12:18:46.219248+0000","last_undegraded":"2024-03-16T12:18:46.219248+0000","last_fullsized":"2024-03-16T12:18:46.219248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.558395+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:18:45.558395+0000","last_peered":"2024-03-16T12:18:45.558395+0000","last_clean":"2024-03-16T12:18:45.558395+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:18:45.558395+0000","last_undegraded":"2024-03-16T12:18:45.558395+0000","last_fullsized":"2024-03-16T12:18:45.558395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.927031+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:06.927031+0000","last_peered":"2024-03-16T12:19:06.927031+0000","last_clean":"2024-03-16T12:19:06.927031+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:06.927031+0000","last_undegraded":"2024-03-16T12:19:06.927031+0000","last_fullsized":"2024-03-16T12:19:06.927031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858395+0000","last_change":"2024-03-16T12:18:44.578288+0000","last_active":"2024-03-16T12:18:46.858395+0000","last_peered":"2024-03-16T12:18:46.858395+0000","last_clean":"2024-03-16T12:18:46.858395+0000","last_became_active":"2024-03-16T12:18:44.577829+0000","last_became_peered":"2024-03-16T12:18:44.577829+0000","last_unstale":"2024-03-16T12:18:46.858395+0000","last_undegraded":"2024-03-16T12:18:46.858395+0000","last_fullsized":"2024-03-16T12:18:46.858395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:40:00.206069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.906806+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:05.906806+0000","last_peered":"2024-03-16T12:19:05.906806+0000","last_clean":"2024-03-16T12:19:05.906806+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:05.906806+0000","last_undegraded":"2024-03-16T12:19:05.906806+0000","last_fullsized":"2024-03-16T12:19:05.906806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559667+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:18:45.559667+0000","last_peered":"2024-03-16T12:18:45.559667+0000","last_clean":"2024-03-16T12:18:45.559667+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:18:45.559667+0000","last_undegraded":"2024-03-16T12:18:45.559667+0000","last_fullsized":"2024-03-16T12:18:45.559667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352263+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:18:46.352263+0000","last_peered":"2024-03-16T12:18:46.352263+0000","last_clean":"2024-03-16T12:18:46.352263+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:18:46.352263+0000","last_undegraded":"2024-03-16T12:18:46.352263+0000","last_fullsized":"2024-03-16T12:18:46.352263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:09.033204+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:19:09.033204+0000","last_peered":"2024-03-16T12:19:09.033204+0000","last_clean":"2024-03-16T12:19:09.033204+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:19:09.033204+0000","last_undegraded":"2024-03-16T12:19:09.033204+0000","last_fullsized":"2024-03-16T12:19:09.033204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353318+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:18:46.353318+0000","last_peered":"2024-03-16T12:18:46.353318+0000","last_clean":"2024-03-16T12:18:46.353318+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:18:46.353318+0000","last_undegraded":"2024-03-16T12:18:46.353318+0000","last_fullsized":"2024-03-16T12:18:46.353318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858641+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:18:46.858641+0000","last_peered":"2024-03-16T12:18:46.858641+0000","last_clean":"2024-03-16T12:18:46.858641+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:18:46.858641+0000","last_undegraded":"2024-03-16T12:18:46.858641+0000","last_fullsized":"2024-03-16T12:18:46.858641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.905505+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:05.905505+0000","last_peered":"2024-03-16T12:19:05.905505+0000","last_clean":"2024-03-16T12:19:05.905505+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:05.905505+0000","last_undegraded":"2024-03-16T12:19:05.905505+0000","last_fullsized":"2024-03-16T12:19:05.905505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.558022+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:18:45.558022+0000","last_peered":"2024-03-16T12:18:45.558022+0000","last_clean":"2024-03-16T12:18:45.558022+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:18:45.558022+0000","last_undegraded":"2024-03-16T12:18:45.558022+0000","last_fullsized":"2024-03-16T12:18:45.558022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858858+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:18:46.858858+0000","last_peered":"2024-03-16T12:18:46.858858+0000","last_clean":"2024-03-16T12:18:46.858858+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:18:46.858858+0000","last_undegraded":"2024-03-16T12:18:46.858858+0000","last_fullsized":"2024-03-16T12:18:46.858858+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.908008+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:05.908008+0000","last_peered":"2024-03-16T12:19:05.908008+0000","last_clean":"2024-03-16T12:19:05.908008+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:05.908008+0000","last_undegraded":"2024-03-16T12:19:05.908008+0000","last_fullsized":"2024-03-16T12:19:05.908008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858354+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:18:46.858354+0000","last_peered":"2024-03-16T12:18:46.858354+0000","last_clean":"2024-03-16T12:18:46.858354+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:18:46.858354+0000","last_undegraded":"2024-03-16T12:18:46.858354+0000","last_fullsized":"2024-03-16T12:18:46.858354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.861336+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:18:46.861336+0000","last_peered":"2024-03-16T12:18:46.861336+0000","last_clean":"2024-03-16T12:18:46.861336+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:18:46.861336+0000","last_undegraded":"2024-03-16T12:18:46.861336+0000","last_fullsized":"2024-03-16T12:18:46.861336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.911444+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:05.911444+0000","last_peered":"2024-03-16T12:19:05.911444+0000","last_clean":"2024-03-16T12:19:05.911444+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:05.911444+0000","last_undegraded":"2024-03-16T12:19:05.911444+0000","last_fullsized":"2024-03-16T12:19:05.911444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858383+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:18:46.858383+0000","last_peered":"2024-03-16T12:18:46.858383+0000","last_clean":"2024-03-16T12:18:46.858383+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:18:46.858383+0000","last_undegraded":"2024-03-16T12:18:46.858383+0000","last_fullsized":"2024-03-16T12:18:46.858383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859184+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:18:46.859184+0000","last_peered":"2024-03-16T12:18:46.859184+0000","last_clean":"2024-03-16T12:18:46.859184+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:18:46.859184+0000","last_undegraded":"2024-03-16T12:18:46.859184+0000","last_fullsized":"2024-03-16T12:18:46.859184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858562+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:18:46.858562+0000","last_peered":"2024-03-16T12:18:46.858562+0000","last_clean":"2024-03-16T12:18:46.858562+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:18:46.858562+0000","last_undegraded":"2024-03-16T12:18:46.858562+0000","last_fullsized":"2024-03-16T12:18:46.858562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352540+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:18:46.352540+0000","last_peered":"2024-03-16T12:18:46.352540+0000","last_clean":"2024-03-16T12:18:46.352540+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:18:46.352540+0000","last_undegraded":"2024-03-16T12:18:46.352540+0000","last_fullsized":"2024-03-16T12:18:46.352540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858791+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:18:46.858791+0000","last_peered":"2024-03-16T12:18:46.858791+0000","last_clean":"2024-03-16T12:18:46.858791+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:18:46.858791+0000","last_undegraded":"2024-03-16T12:18:46.858791+0000","last_fullsized":"2024-03-16T12:18:46.858791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907575+0000","last_change":"2024-03-16T12:18:39.981656+0000","last_active":"2024-03-16T12:19:05.907575+0000","last_peered":"2024-03-16T12:19:05.907575+0000","last_clean":"2024-03-16T12:19:05.907575+0000","last_became_active":"2024-03-16T12:18:39.981300+0000","last_became_peered":"2024-03-16T12:18:39.981300+0000","last_unstale":"2024-03-16T12:19:05.907575+0000","last_undegraded":"2024-03-16T12:19:05.907575+0000","last_fullsized":"2024-03-16T12:19:05.907575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:09.167126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556204+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:18:45.556204+0000","last_peered":"2024-03-16T12:18:45.556204+0000","last_clean":"2024-03-16T12:18:45.556204+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:18:45.556204+0000","last_undegraded":"2024-03-16T12:18:45.556204+0000","last_fullsized":"2024-03-16T12:18:45.556204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858700+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:18:46.858700+0000","last_peered":"2024-03-16T12:18:46.858700+0000","last_clean":"2024-03-16T12:18:46.858700+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:18:46.858700+0000","last_undegraded":"2024-03-16T12:18:46.858700+0000","last_fullsized":"2024-03-16T12:18:46.858700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859423+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:18:46.859423+0000","last_peered":"2024-03-16T12:18:46.859423+0000","last_clean":"2024-03-16T12:18:46.859423+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:18:46.859423+0000","last_undegraded":"2024-03-16T12:18:46.859423+0000","last_fullsized":"2024-03-16T12:18:46.859423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556602+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:18:45.556602+0000","last_peered":"2024-03-16T12:18:45.556602+0000","last_clean":"2024-03-16T12:18:45.556602+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:18:45.556602+0000","last_undegraded":"2024-03-16T12:18:45.556602+0000","last_fullsized":"2024-03-16T12:18:45.556602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556512+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:18:45.556512+0000","last_peered":"2024-03-16T12:18:45.556512+0000","last_clean":"2024-03-16T12:18:45.556512+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:18:45.556512+0000","last_undegraded":"2024-03-16T12:18:45.556512+0000","last_fullsized":"2024-03-16T12:18:45.556512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219302+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:18:46.219302+0000","last_peered":"2024-03-16T12:18:46.219302+0000","last_clean":"2024-03-16T12:18:46.219302+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:18:46.219302+0000","last_undegraded":"2024-03-16T12:18:46.219302+0000","last_fullsized":"2024-03-16T12:18:46.219302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556345+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:18:45.556345+0000","last_peered":"2024-03-16T12:18:45.556345+0000","last_clean":"2024-03-16T12:18:45.556345+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:18:45.556345+0000","last_undegraded":"2024-03-16T12:18:45.556345+0000","last_fullsized":"2024-03-16T12:18:45.556345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218571+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:18:46.218571+0000","last_peered":"2024-03-16T12:18:46.218571+0000","last_clean":"2024-03-16T12:18:46.218571+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:18:46.218571+0000","last_undegraded":"2024-03-16T12:18:46.218571+0000","last_fullsized":"2024-03-16T12:18:46.218571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561255+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:18:45.561255+0000","last_peered":"2024-03-16T12:18:45.561255+0000","last_clean":"2024-03-16T12:18:45.561255+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:18:45.561255+0000","last_undegraded":"2024-03-16T12:18:45.561255+0000","last_fullsized":"2024-03-16T12:18:45.561255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556522+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:18:45.556522+0000","last_peered":"2024-03-16T12:18:45.556522+0000","last_clean":"2024-03-16T12:18:45.556522+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:18:45.556522+0000","last_undegraded":"2024-03-16T12:18:45.556522+0000","last_fullsized":"2024-03-16T12:18:45.556522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858509+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:18:46.858509+0000","last_peered":"2024-03-16T12:18:46.858509+0000","last_clean":"2024-03-16T12:18:46.858509+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:18:46.858509+0000","last_undegraded":"2024-03-16T12:18:46.858509+0000","last_fullsized":"2024-03-16T12:18:46.858509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859648+0000","last_change":"2024-03-16T12:18:40.501669+0000","last_active":"2024-03-16T12:18:46.859648+0000","last_peered":"2024-03-16T12:18:46.859648+0000","last_clean":"2024-03-16T12:18:46.859648+0000","last_became_active":"2024-03-16T12:18:40.499792+0000","last_became_peered":"2024-03-16T12:18:40.499792+0000","last_unstale":"2024-03-16T12:18:46.859648+0000","last_undegraded":"2024-03-16T12:18:46.859648+0000","last_fullsized":"2024-03-16T12:18:46.859648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:36.535106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858449+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:18:46.858449+0000","last_peered":"2024-03-16T12:18:46.858449+0000","last_clean":"2024-03-16T12:18:46.858449+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:18:46.858449+0000","last_undegraded":"2024-03-16T12:18:46.858449+0000","last_fullsized":"2024-03-16T12:18:46.858449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556684+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:18:45.556684+0000","last_peered":"2024-03-16T12:18:45.556684+0000","last_clean":"2024-03-16T12:18:45.556684+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:18:45.556684+0000","last_undegraded":"2024-03-16T12:18:45.556684+0000","last_fullsized":"2024-03-16T12:18:45.556684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559470+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:18:45.559470+0000","last_peered":"2024-03-16T12:18:45.559470+0000","last_clean":"2024-03-16T12:18:45.559470+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:18:45.559470+0000","last_undegraded":"2024-03-16T12:18:45.559470+0000","last_fullsized":"2024-03-16T12:18:45.559470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858801+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:18:46.858801+0000","last_peered":"2024-03-16T12:18:46.858801+0000","last_clean":"2024-03-16T12:18:46.858801+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:18:46.858801+0000","last_undegraded":"2024-03-16T12:18:46.858801+0000","last_fullsized":"2024-03-16T12:18:46.858801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859265+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:18:46.859265+0000","last_peered":"2024-03-16T12:18:46.859265+0000","last_clean":"2024-03-16T12:18:46.859265+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:18:46.859265+0000","last_undegraded":"2024-03-16T12:18:46.859265+0000","last_fullsized":"2024-03-16T12:18:46.859265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353616+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:18:46.353616+0000","last_peered":"2024-03-16T12:18:46.353616+0000","last_clean":"2024-03-16T12:18:46.353616+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:18:46.353616+0000","last_undegraded":"2024-03-16T12:18:46.353616+0000","last_fullsized":"2024-03-16T12:18:46.353616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218827+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:18:46.218827+0000","last_peered":"2024-03-16T12:18:46.218827+0000","last_clean":"2024-03-16T12:18:46.218827+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:18:46.218827+0000","last_undegraded":"2024-03-16T12:18:46.218827+0000","last_fullsized":"2024-03-16T12:18:46.218827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353267+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:18:46.353267+0000","last_peered":"2024-03-16T12:18:46.353267+0000","last_clean":"2024-03-16T12:18:46.353267+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:18:46.353267+0000","last_undegraded":"2024-03-16T12:18:46.353267+0000","last_fullsized":"2024-03-16T12:18:46.353267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858974+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:18:46.858974+0000","last_peered":"2024-03-16T12:18:46.858974+0000","last_clean":"2024-03-16T12:18:46.858974+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:18:46.858974+0000","last_undegraded":"2024-03-16T12:18:46.858974+0000","last_fullsized":"2024-03-16T12:18:46.858974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559306+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:18:45.559306+0000","last_peered":"2024-03-16T12:18:45.559306+0000","last_clean":"2024-03-16T12:18:45.559306+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:18:45.559306+0000","last_undegraded":"2024-03-16T12:18:45.559306+0000","last_fullsized":"2024-03-16T12:18:45.559306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859293+0000","last_change":"2024-03-16T12:18:40.502717+0000","last_active":"2024-03-16T12:18:46.859293+0000","last_peered":"2024-03-16T12:18:46.859293+0000","last_clean":"2024-03-16T12:18:46.859293+0000","last_became_active":"2024-03-16T12:18:40.502372+0000","last_became_peered":"2024-03-16T12:18:40.502372+0000","last_unstale":"2024-03-16T12:18:46.859293+0000","last_undegraded":"2024-03-16T12:18:46.859293+0000","last_fullsized":"2024-03-16T12:18:46.859293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:31:23.058685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6870,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":121,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":18940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1597440,"data_stored":1073471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495662,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1609728,"data_stored":1073155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:19:11.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T12:19:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 -- 172.21.15.98:0/1969923942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 msgr2=0x7f91ec109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1969923942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec109470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009900 tx=0x7f91d402bc40 comp rx=0 tx=0).stop 2024-03-16T12:19:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 -- 172.21.15.98:0/1969923942 shutdown_connections 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1969923942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1969923942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec109470 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1969923942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 0x7f91ec108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 -- 172.21.15.98:0/1969923942 >> 172.21.15.98:0/1969923942 conn(0x7f91ec076dc0 msgr2=0x7f91ec0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 -- 172.21.15.98:0/1969923942 shutdown_connections 2024-03-16T12:19:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.623+0000 7f91f2c00640 1 -- 172.21.15.98:0/1969923942 wait complete. 2024-03-16T12:19:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 Processor -- start 2024-03-16T12:19:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 -- start start 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 0x7f91ec19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec113ad0 con 0x7f91ec107e00 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec113950 con 0x7f91ec1099b0 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f2c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec113c50 con 0x7f91ec108ff0 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91ebe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 0x7f91ec19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:47924/0 (socket says 172.21.15.98:47924) 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91f0c00640 1 -- 172.21.15.98:0/1355042134 learned_addr learned my addr 172.21.15.98:0/1355042134 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.624+0000 7f91e3400640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 -- 172.21.15.98:0/1355042134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 msgr2=0x7f91ec19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 -- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 msgr2=0x7f91ec19e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 0x7f91ec19e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 -- 172.21.15.98:0/1355042134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec1a5d60 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f0c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f91e400eb10 tx=0x7f91e400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4004b20 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e4004cc0 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1a5ff0 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.625+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4010760 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.626+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1a6560 con 0x7f91ec1099b0 2024-03-16T12:19:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.627+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec10f0c0 con 0x7f91ec1099b0 2024-03-16T12:19:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.629+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f91e40459e0 con 0x7f91ec1099b0 2024-03-16T12:19:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.630+0000 7f91e9600640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.630+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f91e40cf480 con 0x7f91ec1099b0 2024-03-16T12:19:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.630+0000 7f91ebe00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.631+0000 7f91ebe00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 0x7f91d00a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91ec108e50 tx=0x7f91dc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.636+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91e409c180 con 0x7f91ec1099b0 2024-03-16T12:19:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.843+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f91ec061c20 con 0x7f91d00a2910 2024-03-16T12:19:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.858+0000 7f91e9600640 1 -- 172.21.15.98:0/1355042134 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340362 (secure 0 0 0) 0x7f91ec061c20 con 0x7f91d00a2910 2024-03-16T12:19:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 msgr2=0x7f91d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 0x7f91d00a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91ec108e50 tx=0x7f91dc00b390 comp rx=0 tx=0).stop 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 msgr2=0x7f91ec1a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f91e400eb10 tx=0x7f91e400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 shutdown_connections 2024-03-16T12:19:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec1099b0 0x7f91ec1a56f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91ec108ff0 0x7f91ec19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 --2- 172.21.15.98:0/1355042134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec107e00 0x7f91ec19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 >> 172.21.15.98:0/1355042134 conn(0x7f91ec076dc0 msgr2=0x7f91ec10c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 shutdown_connections 2024-03-16T12:19:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:11.865+0000 7f91f2c00640 1 -- 172.21.15.98:0/1355042134 wait complete. 2024-03-16T12:19:11.865 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-16T12:19:10.780377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597355,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":148,"num_read_kb":127,"num_write":303,"num_write_kb":2984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383424,"kb_used_data":13460,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558400,"statfs":{"total":1151940427776,"available":1151547801600,"internally_reserved":0,"allocated":13783040,"data_stored":7533008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4128,"num_objects":33,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":76,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005417"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859526+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:18:46.859526+0000","last_peered":"2024-03-16T12:18:46.859526+0000","last_clean":"2024-03-16T12:18:46.859526+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:18:46.859526+0000","last_undegraded":"2024-03-16T12:18:46.859526+0000","last_fullsized":"2024-03-16T12:18:46.859526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557727+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:18:45.557727+0000","last_peered":"2024-03-16T12:18:45.557727+0000","last_clean":"2024-03-16T12:18:45.557727+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:18:45.557727+0000","last_undegraded":"2024-03-16T12:18:45.557727+0000","last_fullsized":"2024-03-16T12:18:45.557727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557290+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:18:45.557290+0000","last_peered":"2024-03-16T12:18:45.557290+0000","last_clean":"2024-03-16T12:18:45.557290+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:18:45.557290+0000","last_undegraded":"2024-03-16T12:18:45.557290+0000","last_fullsized":"2024-03-16T12:18:45.557290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859435+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:18:46.859435+0000","last_peered":"2024-03-16T12:18:46.859435+0000","last_clean":"2024-03-16T12:18:46.859435+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:18:46.859435+0000","last_undegraded":"2024-03-16T12:18:46.859435+0000","last_fullsized":"2024-03-16T12:18:46.859435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859032+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:18:46.859032+0000","last_peered":"2024-03-16T12:18:46.859032+0000","last_clean":"2024-03-16T12:18:46.859032+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:18:46.859032+0000","last_undegraded":"2024-03-16T12:18:46.859032+0000","last_fullsized":"2024-03-16T12:18:46.859032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858382+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:18:46.858382+0000","last_peered":"2024-03-16T12:18:46.858382+0000","last_clean":"2024-03-16T12:18:46.858382+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:18:46.858382+0000","last_undegraded":"2024-03-16T12:18:46.858382+0000","last_fullsized":"2024-03-16T12:18:46.858382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556016+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:18:45.556016+0000","last_peered":"2024-03-16T12:18:45.556016+0000","last_clean":"2024-03-16T12:18:45.556016+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:18:45.556016+0000","last_undegraded":"2024-03-16T12:18:45.556016+0000","last_fullsized":"2024-03-16T12:18:45.556016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.351902+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:18:46.351902+0000","last_peered":"2024-03-16T12:18:46.351902+0000","last_clean":"2024-03-16T12:18:46.351902+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:18:46.351902+0000","last_undegraded":"2024-03-16T12:18:46.351902+0000","last_fullsized":"2024-03-16T12:18:46.351902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859210+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:18:46.859210+0000","last_peered":"2024-03-16T12:18:46.859210+0000","last_clean":"2024-03-16T12:18:46.859210+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:18:46.859210+0000","last_undegraded":"2024-03-16T12:18:46.859210+0000","last_fullsized":"2024-03-16T12:18:46.859210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858497+0000","last_change":"2024-03-16T12:18:40.503141+0000","last_active":"2024-03-16T12:18:46.858497+0000","last_peered":"2024-03-16T12:18:46.858497+0000","last_clean":"2024-03-16T12:18:46.858497+0000","last_became_active":"2024-03-16T12:18:40.502789+0000","last_became_peered":"2024-03-16T12:18:40.502789+0000","last_unstale":"2024-03-16T12:18:46.858497+0000","last_undegraded":"2024-03-16T12:18:46.858497+0000","last_fullsized":"2024-03-16T12:18:46.858497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:37.220085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556575+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:18:45.556575+0000","last_peered":"2024-03-16T12:18:45.556575+0000","last_clean":"2024-03-16T12:18:45.556575+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:18:45.556575+0000","last_undegraded":"2024-03-16T12:18:45.556575+0000","last_fullsized":"2024-03-16T12:18:45.556575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859352+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:18:46.859352+0000","last_peered":"2024-03-16T12:18:46.859352+0000","last_clean":"2024-03-16T12:18:46.859352+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:18:46.859352+0000","last_undegraded":"2024-03-16T12:18:46.859352+0000","last_fullsized":"2024-03-16T12:18:46.859352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559046+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:18:45.559046+0000","last_peered":"2024-03-16T12:18:45.559046+0000","last_clean":"2024-03-16T12:18:45.559046+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:18:45.559046+0000","last_undegraded":"2024-03-16T12:18:45.559046+0000","last_fullsized":"2024-03-16T12:18:45.559046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556137+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:18:45.556137+0000","last_peered":"2024-03-16T12:18:45.556137+0000","last_clean":"2024-03-16T12:18:45.556137+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:18:45.556137+0000","last_undegraded":"2024-03-16T12:18:45.556137+0000","last_fullsized":"2024-03-16T12:18:45.556137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556398+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:18:45.556398+0000","last_peered":"2024-03-16T12:18:45.556398+0000","last_clean":"2024-03-16T12:18:45.556398+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:18:45.556398+0000","last_undegraded":"2024-03-16T12:18:45.556398+0000","last_fullsized":"2024-03-16T12:18:45.556398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858431+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:18:46.858431+0000","last_peered":"2024-03-16T12:18:46.858431+0000","last_clean":"2024-03-16T12:18:46.858431+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:18:46.858431+0000","last_undegraded":"2024-03-16T12:18:46.858431+0000","last_fullsized":"2024-03-16T12:18:46.858431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859429+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:18:46.859429+0000","last_peered":"2024-03-16T12:18:46.859429+0000","last_clean":"2024-03-16T12:18:46.859429+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:18:46.859429+0000","last_undegraded":"2024-03-16T12:18:46.859429+0000","last_fullsized":"2024-03-16T12:18:46.859429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.861061+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:18:46.861061+0000","last_peered":"2024-03-16T12:18:46.861061+0000","last_clean":"2024-03-16T12:18:46.861061+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:18:46.861061+0000","last_undegraded":"2024-03-16T12:18:46.861061+0000","last_fullsized":"2024-03-16T12:18:46.861061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858863+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:18:46.858863+0000","last_peered":"2024-03-16T12:18:46.858863+0000","last_clean":"2024-03-16T12:18:46.858863+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:18:46.858863+0000","last_undegraded":"2024-03-16T12:18:46.858863+0000","last_fullsized":"2024-03-16T12:18:46.858863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859569+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:18:46.859569+0000","last_peered":"2024-03-16T12:18:46.859569+0000","last_clean":"2024-03-16T12:18:46.859569+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:18:46.859569+0000","last_undegraded":"2024-03-16T12:18:46.859569+0000","last_fullsized":"2024-03-16T12:18:46.859569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.932837+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:06.932837+0000","last_peered":"2024-03-16T12:19:06.932837+0000","last_clean":"2024-03-16T12:19:06.932837+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:06.932837+0000","last_undegraded":"2024-03-16T12:19:06.932837+0000","last_fullsized":"2024-03-16T12:19:06.932837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352729+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:18:46.352729+0000","last_peered":"2024-03-16T12:18:46.352729+0000","last_clean":"2024-03-16T12:18:46.352729+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:18:46.352729+0000","last_undegraded":"2024-03-16T12:18:46.352729+0000","last_fullsized":"2024-03-16T12:18:46.352729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.857987+0000","last_change":"2024-03-16T12:18:44.590453+0000","last_active":"2024-03-16T12:18:46.857987+0000","last_peered":"2024-03-16T12:18:46.857987+0000","last_clean":"2024-03-16T12:18:46.857987+0000","last_became_active":"2024-03-16T12:18:44.589804+0000","last_became_peered":"2024-03-16T12:18:44.589804+0000","last_unstale":"2024-03-16T12:18:46.857987+0000","last_undegraded":"2024-03-16T12:18:46.857987+0000","last_fullsized":"2024-03-16T12:18:46.857987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:05.144303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353104+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:18:46.353104+0000","last_peered":"2024-03-16T12:18:46.353104+0000","last_clean":"2024-03-16T12:18:46.353104+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:18:46.353104+0000","last_undegraded":"2024-03-16T12:18:46.353104+0000","last_fullsized":"2024-03-16T12:18:46.353104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557830+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:18:45.557830+0000","last_peered":"2024-03-16T12:18:45.557830+0000","last_clean":"2024-03-16T12:18:45.557830+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:18:45.557830+0000","last_undegraded":"2024-03-16T12:18:45.557830+0000","last_fullsized":"2024-03-16T12:18:45.557830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859516+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:18:46.859516+0000","last_peered":"2024-03-16T12:18:46.859516+0000","last_clean":"2024-03-16T12:18:46.859516+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:18:46.859516+0000","last_undegraded":"2024-03-16T12:18:46.859516+0000","last_fullsized":"2024-03-16T12:18:46.859516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.927222+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:06.927222+0000","last_peered":"2024-03-16T12:19:06.927222+0000","last_clean":"2024-03-16T12:19:06.927222+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:06.927222+0000","last_undegraded":"2024-03-16T12:19:06.927222+0000","last_fullsized":"2024-03-16T12:19:06.927222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858760+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:18:46.858760+0000","last_peered":"2024-03-16T12:18:46.858760+0000","last_clean":"2024-03-16T12:18:46.858760+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:18:46.858760+0000","last_undegraded":"2024-03-16T12:18:46.858760+0000","last_fullsized":"2024-03-16T12:18:46.858760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858351+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:18:46.858351+0000","last_peered":"2024-03-16T12:18:46.858351+0000","last_clean":"2024-03-16T12:18:46.858351+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:18:46.858351+0000","last_undegraded":"2024-03-16T12:18:46.858351+0000","last_fullsized":"2024-03-16T12:18:46.858351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859795+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:18:46.859795+0000","last_peered":"2024-03-16T12:18:46.859795+0000","last_clean":"2024-03-16T12:18:46.859795+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:18:46.859795+0000","last_undegraded":"2024-03-16T12:18:46.859795+0000","last_fullsized":"2024-03-16T12:18:46.859795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219215+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:18:46.219215+0000","last_peered":"2024-03-16T12:18:46.219215+0000","last_clean":"2024-03-16T12:18:46.219215+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:18:46.219215+0000","last_undegraded":"2024-03-16T12:18:46.219215+0000","last_fullsized":"2024-03-16T12:18:46.219215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858856+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:18:46.858856+0000","last_peered":"2024-03-16T12:18:46.858856+0000","last_clean":"2024-03-16T12:18:46.858856+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:18:46.858856+0000","last_undegraded":"2024-03-16T12:18:46.858856+0000","last_fullsized":"2024-03-16T12:18:46.858856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.489025+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:19:07.489025+0000","last_peered":"2024-03-16T12:19:07.489025+0000","last_clean":"2024-03-16T12:19:07.489025+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:19:07.489025+0000","last_undegraded":"2024-03-16T12:19:07.489025+0000","last_fullsized":"2024-03-16T12:19:07.489025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858739+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:18:46.858739+0000","last_peered":"2024-03-16T12:18:46.858739+0000","last_clean":"2024-03-16T12:18:46.858739+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:18:46.858739+0000","last_undegraded":"2024-03-16T12:18:46.858739+0000","last_fullsized":"2024-03-16T12:18:46.858739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859150+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:18:46.859150+0000","last_peered":"2024-03-16T12:18:46.859150+0000","last_clean":"2024-03-16T12:18:46.859150+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:18:46.859150+0000","last_undegraded":"2024-03-16T12:18:46.859150+0000","last_fullsized":"2024-03-16T12:18:46.859150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858356+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:18:46.858356+0000","last_peered":"2024-03-16T12:18:46.858356+0000","last_clean":"2024-03-16T12:18:46.858356+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:18:46.858356+0000","last_undegraded":"2024-03-16T12:18:46.858356+0000","last_fullsized":"2024-03-16T12:18:46.858356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219265+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:18:46.219265+0000","last_peered":"2024-03-16T12:18:46.219265+0000","last_clean":"2024-03-16T12:18:46.219265+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:18:46.219265+0000","last_undegraded":"2024-03-16T12:18:46.219265+0000","last_fullsized":"2024-03-16T12:18:46.219265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859712+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:18:46.859712+0000","last_peered":"2024-03-16T12:18:46.859712+0000","last_clean":"2024-03-16T12:18:46.859712+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:18:46.859712+0000","last_undegraded":"2024-03-16T12:18:46.859712+0000","last_fullsized":"2024-03-16T12:18:46.859712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.933723+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:06.933723+0000","last_peered":"2024-03-16T12:19:06.933723+0000","last_clean":"2024-03-16T12:19:06.933723+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:06.933723+0000","last_undegraded":"2024-03-16T12:19:06.933723+0000","last_fullsized":"2024-03-16T12:19:06.933723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352163+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:18:46.352163+0000","last_peered":"2024-03-16T12:18:46.352163+0000","last_clean":"2024-03-16T12:18:46.352163+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:18:46.352163+0000","last_undegraded":"2024-03-16T12:18:46.352163+0000","last_fullsized":"2024-03-16T12:18:46.352163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557872+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:18:45.557872+0000","last_peered":"2024-03-16T12:18:45.557872+0000","last_clean":"2024-03-16T12:18:45.557872+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:18:45.557872+0000","last_undegraded":"2024-03-16T12:18:45.557872+0000","last_fullsized":"2024-03-16T12:18:45.557872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218865+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:18:46.218865+0000","last_peered":"2024-03-16T12:18:46.218865+0000","last_clean":"2024-03-16T12:18:46.218865+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:18:46.218865+0000","last_undegraded":"2024-03-16T12:18:46.218865+0000","last_fullsized":"2024-03-16T12:18:46.218865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859609+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:18:46.859609+0000","last_peered":"2024-03-16T12:18:46.859609+0000","last_clean":"2024-03-16T12:18:46.859609+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:18:46.859609+0000","last_undegraded":"2024-03-16T12:18:46.859609+0000","last_fullsized":"2024-03-16T12:18:46.859609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352626+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:18:46.352626+0000","last_peered":"2024-03-16T12:18:46.352626+0000","last_clean":"2024-03-16T12:18:46.352626+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:18:46.352626+0000","last_undegraded":"2024-03-16T12:18:46.352626+0000","last_fullsized":"2024-03-16T12:18:46.352626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907419+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:19:05.907419+0000","last_peered":"2024-03-16T12:19:05.907419+0000","last_clean":"2024-03-16T12:19:05.907419+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:19:05.907419+0000","last_undegraded":"2024-03-16T12:19:05.907419+0000","last_fullsized":"2024-03-16T12:19:05.907419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858813+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:18:46.858813+0000","last_peered":"2024-03-16T12:18:46.858813+0000","last_clean":"2024-03-16T12:18:46.858813+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:18:46.858813+0000","last_undegraded":"2024-03-16T12:18:46.858813+0000","last_fullsized":"2024-03-16T12:18:46.858813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859796+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:18:46.859796+0000","last_peered":"2024-03-16T12:18:46.859796+0000","last_clean":"2024-03-16T12:18:46.859796+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:18:46.859796+0000","last_undegraded":"2024-03-16T12:18:46.859796+0000","last_fullsized":"2024-03-16T12:18:46.859796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.906433+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:05.906433+0000","last_peered":"2024-03-16T12:19:05.906433+0000","last_clean":"2024-03-16T12:19:05.906433+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:05.906433+0000","last_undegraded":"2024-03-16T12:19:05.906433+0000","last_fullsized":"2024-03-16T12:19:05.906433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.557834+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:18:45.557834+0000","last_peered":"2024-03-16T12:18:45.557834+0000","last_clean":"2024-03-16T12:18:45.557834+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:18:45.557834+0000","last_undegraded":"2024-03-16T12:18:45.557834+0000","last_fullsized":"2024-03-16T12:18:45.557834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859678+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:18:46.859678+0000","last_peered":"2024-03-16T12:18:46.859678+0000","last_clean":"2024-03-16T12:18:46.859678+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:18:46.859678+0000","last_undegraded":"2024-03-16T12:18:46.859678+0000","last_fullsized":"2024-03-16T12:18:46.859678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907938+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:05.907938+0000","last_peered":"2024-03-16T12:19:05.907938+0000","last_clean":"2024-03-16T12:19:05.907938+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:05.907938+0000","last_undegraded":"2024-03-16T12:19:05.907938+0000","last_fullsized":"2024-03-16T12:19:05.907938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858941+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:18:46.858941+0000","last_peered":"2024-03-16T12:18:46.858941+0000","last_clean":"2024-03-16T12:18:46.858941+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:18:46.858941+0000","last_undegraded":"2024-03-16T12:18:46.858941+0000","last_fullsized":"2024-03-16T12:18:46.858941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859732+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:18:46.859732+0000","last_peered":"2024-03-16T12:18:46.859732+0000","last_clean":"2024-03-16T12:18:46.859732+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:18:46.859732+0000","last_undegraded":"2024-03-16T12:18:46.859732+0000","last_fullsized":"2024-03-16T12:18:46.859732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.905660+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:19:05.905660+0000","last_peered":"2024-03-16T12:19:05.905660+0000","last_clean":"2024-03-16T12:19:05.905660+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:19:05.905660+0000","last_undegraded":"2024-03-16T12:19:05.905660+0000","last_fullsized":"2024-03-16T12:19:05.905660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858919+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:18:46.858919+0000","last_peered":"2024-03-16T12:18:46.858919+0000","last_clean":"2024-03-16T12:18:46.858919+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:18:46.858919+0000","last_undegraded":"2024-03-16T12:18:46.858919+0000","last_fullsized":"2024-03-16T12:18:46.858919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219498+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:18:46.219498+0000","last_peered":"2024-03-16T12:18:46.219498+0000","last_clean":"2024-03-16T12:18:46.219498+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:18:46.219498+0000","last_undegraded":"2024-03-16T12:18:46.219498+0000","last_fullsized":"2024-03-16T12:18:46.219498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859382+0000","last_change":"2024-03-16T12:18:39.988134+0000","last_active":"2024-03-16T12:18:46.859382+0000","last_peered":"2024-03-16T12:18:46.859382+0000","last_clean":"2024-03-16T12:18:46.859382+0000","last_became_active":"2024-03-16T12:18:39.987772+0000","last_became_peered":"2024-03-16T12:18:39.987772+0000","last_unstale":"2024-03-16T12:18:46.859382+0000","last_undegraded":"2024-03-16T12:18:46.859382+0000","last_fullsized":"2024-03-16T12:18:46.859382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:49.889547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559808+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:18:45.559808+0000","last_peered":"2024-03-16T12:18:45.559808+0000","last_clean":"2024-03-16T12:18:45.559808+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:18:45.559808+0000","last_undegraded":"2024-03-16T12:18:45.559808+0000","last_fullsized":"2024-03-16T12:18:45.559808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859114+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:18:46.859114+0000","last_peered":"2024-03-16T12:18:46.859114+0000","last_clean":"2024-03-16T12:18:46.859114+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:18:46.859114+0000","last_undegraded":"2024-03-16T12:18:46.859114+0000","last_fullsized":"2024-03-16T12:18:46.859114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907823+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:05.907823+0000","last_peered":"2024-03-16T12:19:05.907823+0000","last_clean":"2024-03-16T12:19:05.907823+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:05.907823+0000","last_undegraded":"2024-03-16T12:19:05.907823+0000","last_fullsized":"2024-03-16T12:19:05.907823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556262+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:18:45.556262+0000","last_peered":"2024-03-16T12:18:45.556262+0000","last_clean":"2024-03-16T12:18:45.556262+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:18:45.556262+0000","last_undegraded":"2024-03-16T12:18:45.556262+0000","last_fullsized":"2024-03-16T12:18:45.556262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.860624+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:18:46.860624+0000","last_peered":"2024-03-16T12:18:46.860624+0000","last_clean":"2024-03-16T12:18:46.860624+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:18:46.860624+0000","last_undegraded":"2024-03-16T12:18:46.860624+0000","last_fullsized":"2024-03-16T12:18:46.860624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.490502+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:19:07.490502+0000","last_peered":"2024-03-16T12:19:07.490502+0000","last_clean":"2024-03-16T12:19:07.490502+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:19:07.490502+0000","last_undegraded":"2024-03-16T12:19:07.490502+0000","last_fullsized":"2024-03-16T12:19:07.490502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556058+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:18:45.556058+0000","last_peered":"2024-03-16T12:18:45.556058+0000","last_clean":"2024-03-16T12:18:45.556058+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:18:45.556058+0000","last_undegraded":"2024-03-16T12:18:45.556058+0000","last_fullsized":"2024-03-16T12:18:45.556058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561332+0000","last_change":"2024-03-16T12:18:44.589442+0000","last_active":"2024-03-16T12:18:45.561332+0000","last_peered":"2024-03-16T12:18:45.561332+0000","last_clean":"2024-03-16T12:18:45.561332+0000","last_became_active":"2024-03-16T12:18:44.588933+0000","last_became_peered":"2024-03-16T12:18:44.588933+0000","last_unstale":"2024-03-16T12:18:45.561332+0000","last_undegraded":"2024-03-16T12:18:45.561332+0000","last_fullsized":"2024-03-16T12:18:45.561332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:07:10.860786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.934282+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:06.934282+0000","last_peered":"2024-03-16T12:19:06.934282+0000","last_clean":"2024-03-16T12:19:06.934282+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:06.934282+0000","last_undegraded":"2024-03-16T12:19:06.934282+0000","last_fullsized":"2024-03-16T12:19:06.934282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218791+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:18:46.218791+0000","last_peered":"2024-03-16T12:18:46.218791+0000","last_clean":"2024-03-16T12:18:46.218791+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:18:46.218791+0000","last_undegraded":"2024-03-16T12:18:46.218791+0000","last_fullsized":"2024-03-16T12:18:46.218791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218822+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:18:46.218822+0000","last_peered":"2024-03-16T12:18:46.218822+0000","last_clean":"2024-03-16T12:18:46.218822+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:18:46.218822+0000","last_undegraded":"2024-03-16T12:18:46.218822+0000","last_fullsized":"2024-03-16T12:18:46.218822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.486344+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:07.486344+0000","last_peered":"2024-03-16T12:19:07.486344+0000","last_clean":"2024-03-16T12:19:07.486344+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:07.486344+0000","last_undegraded":"2024-03-16T12:19:07.486344+0000","last_fullsized":"2024-03-16T12:19:07.486344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858850+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:18:46.858850+0000","last_peered":"2024-03-16T12:18:46.858850+0000","last_clean":"2024-03-16T12:18:46.858850+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:18:46.858850+0000","last_undegraded":"2024-03-16T12:18:46.858850+0000","last_fullsized":"2024-03-16T12:18:46.858850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858623+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:18:46.858623+0000","last_peered":"2024-03-16T12:18:46.858623+0000","last_clean":"2024-03-16T12:18:46.858623+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:18:46.858623+0000","last_undegraded":"2024-03-16T12:18:46.858623+0000","last_fullsized":"2024-03-16T12:18:46.858623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.860566+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:18:46.860566+0000","last_peered":"2024-03-16T12:18:46.860566+0000","last_clean":"2024-03-16T12:18:46.860566+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:18:46.860566+0000","last_undegraded":"2024-03-16T12:18:46.860566+0000","last_fullsized":"2024-03-16T12:18:46.860566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858658+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:18:46.858658+0000","last_peered":"2024-03-16T12:18:46.858658+0000","last_clean":"2024-03-16T12:18:46.858658+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:18:46.858658+0000","last_undegraded":"2024-03-16T12:18:46.858658+0000","last_fullsized":"2024-03-16T12:18:46.858658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559274+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:18:45.559274+0000","last_peered":"2024-03-16T12:18:45.559274+0000","last_clean":"2024-03-16T12:18:45.559274+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:18:45.559274+0000","last_undegraded":"2024-03-16T12:18:45.559274+0000","last_fullsized":"2024-03-16T12:18:45.559274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556218+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:18:45.556218+0000","last_peered":"2024-03-16T12:18:45.556218+0000","last_clean":"2024-03-16T12:18:45.556218+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:18:45.556218+0000","last_undegraded":"2024-03-16T12:18:45.556218+0000","last_fullsized":"2024-03-16T12:18:45.556218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858105+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:18:46.858105+0000","last_peered":"2024-03-16T12:18:46.858105+0000","last_clean":"2024-03-16T12:18:46.858105+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:18:46.858105+0000","last_undegraded":"2024-03-16T12:18:46.858105+0000","last_fullsized":"2024-03-16T12:18:46.858105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561198+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:18:45.561198+0000","last_peered":"2024-03-16T12:18:45.561198+0000","last_clean":"2024-03-16T12:18:45.561198+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:18:45.561198+0000","last_undegraded":"2024-03-16T12:18:45.561198+0000","last_fullsized":"2024-03-16T12:18:45.561198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218929+0000","last_change":"2024-03-16T12:18:40.505877+0000","last_active":"2024-03-16T12:18:46.218929+0000","last_peered":"2024-03-16T12:18:46.218929+0000","last_clean":"2024-03-16T12:18:46.218929+0000","last_became_active":"2024-03-16T12:18:40.504655+0000","last_became_peered":"2024-03-16T12:18:40.504655+0000","last_unstale":"2024-03-16T12:18:46.218929+0000","last_undegraded":"2024-03-16T12:18:46.218929+0000","last_fullsized":"2024-03-16T12:18:46.218929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:27.043333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.488421+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:19:07.488421+0000","last_peered":"2024-03-16T12:19:07.488421+0000","last_clean":"2024-03-16T12:19:07.488421+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:19:07.488421+0000","last_undegraded":"2024-03-16T12:19:07.488421+0000","last_fullsized":"2024-03-16T12:19:07.488421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556553+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:18:45.556553+0000","last_peered":"2024-03-16T12:18:45.556553+0000","last_clean":"2024-03-16T12:18:45.556553+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:18:45.556553+0000","last_undegraded":"2024-03-16T12:18:45.556553+0000","last_fullsized":"2024-03-16T12:18:45.556553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352435+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:18:46.352435+0000","last_peered":"2024-03-16T12:18:46.352435+0000","last_clean":"2024-03-16T12:18:46.352435+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:18:46.352435+0000","last_undegraded":"2024-03-16T12:18:46.352435+0000","last_fullsized":"2024-03-16T12:18:46.352435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:07.485920+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:07.485920+0000","last_peered":"2024-03-16T12:19:07.485920+0000","last_clean":"2024-03-16T12:19:07.485920+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:07.485920+0000","last_undegraded":"2024-03-16T12:19:07.485920+0000","last_fullsized":"2024-03-16T12:19:07.485920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.555947+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:18:45.555947+0000","last_peered":"2024-03-16T12:18:45.555947+0000","last_clean":"2024-03-16T12:18:45.555947+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:18:45.555947+0000","last_undegraded":"2024-03-16T12:18:45.555947+0000","last_fullsized":"2024-03-16T12:18:45.555947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219248+0000","last_change":"2024-03-16T12:18:44.588798+0000","last_active":"2024-03-16T12:18:46.219248+0000","last_peered":"2024-03-16T12:18:46.219248+0000","last_clean":"2024-03-16T12:18:46.219248+0000","last_became_active":"2024-03-16T12:18:44.588498+0000","last_became_peered":"2024-03-16T12:18:44.588498+0000","last_unstale":"2024-03-16T12:18:46.219248+0000","last_undegraded":"2024-03-16T12:18:46.219248+0000","last_fullsized":"2024-03-16T12:18:46.219248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.558395+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:18:45.558395+0000","last_peered":"2024-03-16T12:18:45.558395+0000","last_clean":"2024-03-16T12:18:45.558395+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:18:45.558395+0000","last_undegraded":"2024-03-16T12:18:45.558395+0000","last_fullsized":"2024-03-16T12:18:45.558395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:06.927031+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:06.927031+0000","last_peered":"2024-03-16T12:19:06.927031+0000","last_clean":"2024-03-16T12:19:06.927031+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:06.927031+0000","last_undegraded":"2024-03-16T12:19:06.927031+0000","last_fullsized":"2024-03-16T12:19:06.927031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858395+0000","last_change":"2024-03-16T12:18:44.578288+0000","last_active":"2024-03-16T12:18:46.858395+0000","last_peered":"2024-03-16T12:18:46.858395+0000","last_clean":"2024-03-16T12:18:46.858395+0000","last_became_active":"2024-03-16T12:18:44.577829+0000","last_became_peered":"2024-03-16T12:18:44.577829+0000","last_unstale":"2024-03-16T12:18:46.858395+0000","last_undegraded":"2024-03-16T12:18:46.858395+0000","last_fullsized":"2024-03-16T12:18:46.858395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:40:00.206069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.906806+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:05.906806+0000","last_peered":"2024-03-16T12:19:05.906806+0000","last_clean":"2024-03-16T12:19:05.906806+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:05.906806+0000","last_undegraded":"2024-03-16T12:19:05.906806+0000","last_fullsized":"2024-03-16T12:19:05.906806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559667+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:18:45.559667+0000","last_peered":"2024-03-16T12:18:45.559667+0000","last_clean":"2024-03-16T12:18:45.559667+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:18:45.559667+0000","last_undegraded":"2024-03-16T12:18:45.559667+0000","last_fullsized":"2024-03-16T12:18:45.559667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352263+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:18:46.352263+0000","last_peered":"2024-03-16T12:18:46.352263+0000","last_clean":"2024-03-16T12:18:46.352263+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:18:46.352263+0000","last_undegraded":"2024-03-16T12:18:46.352263+0000","last_fullsized":"2024-03-16T12:18:46.352263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:09.033204+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:19:09.033204+0000","last_peered":"2024-03-16T12:19:09.033204+0000","last_clean":"2024-03-16T12:19:09.033204+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:19:09.033204+0000","last_undegraded":"2024-03-16T12:19:09.033204+0000","last_fullsized":"2024-03-16T12:19:09.033204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353318+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:18:46.353318+0000","last_peered":"2024-03-16T12:18:46.353318+0000","last_clean":"2024-03-16T12:18:46.353318+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:18:46.353318+0000","last_undegraded":"2024-03-16T12:18:46.353318+0000","last_fullsized":"2024-03-16T12:18:46.353318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858641+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:18:46.858641+0000","last_peered":"2024-03-16T12:18:46.858641+0000","last_clean":"2024-03-16T12:18:46.858641+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:18:46.858641+0000","last_undegraded":"2024-03-16T12:18:46.858641+0000","last_fullsized":"2024-03-16T12:18:46.858641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.905505+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:05.905505+0000","last_peered":"2024-03-16T12:19:05.905505+0000","last_clean":"2024-03-16T12:19:05.905505+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:05.905505+0000","last_undegraded":"2024-03-16T12:19:05.905505+0000","last_fullsized":"2024-03-16T12:19:05.905505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.558022+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:18:45.558022+0000","last_peered":"2024-03-16T12:18:45.558022+0000","last_clean":"2024-03-16T12:18:45.558022+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:18:45.558022+0000","last_undegraded":"2024-03-16T12:18:45.558022+0000","last_fullsized":"2024-03-16T12:18:45.558022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858858+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:18:46.858858+0000","last_peered":"2024-03-16T12:18:46.858858+0000","last_clean":"2024-03-16T12:18:46.858858+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:18:46.858858+0000","last_undegraded":"2024-03-16T12:18:46.858858+0000","last_fullsized":"2024-03-16T12:18:46.858858+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.908008+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:05.908008+0000","last_peered":"2024-03-16T12:19:05.908008+0000","last_clean":"2024-03-16T12:19:05.908008+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:05.908008+0000","last_undegraded":"2024-03-16T12:19:05.908008+0000","last_fullsized":"2024-03-16T12:19:05.908008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858354+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:18:46.858354+0000","last_peered":"2024-03-16T12:18:46.858354+0000","last_clean":"2024-03-16T12:18:46.858354+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:18:46.858354+0000","last_undegraded":"2024-03-16T12:18:46.858354+0000","last_fullsized":"2024-03-16T12:18:46.858354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.861336+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:18:46.861336+0000","last_peered":"2024-03-16T12:18:46.861336+0000","last_clean":"2024-03-16T12:18:46.861336+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:18:46.861336+0000","last_undegraded":"2024-03-16T12:18:46.861336+0000","last_fullsized":"2024-03-16T12:18:46.861336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.911444+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:05.911444+0000","last_peered":"2024-03-16T12:19:05.911444+0000","last_clean":"2024-03-16T12:19:05.911444+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:05.911444+0000","last_undegraded":"2024-03-16T12:19:05.911444+0000","last_fullsized":"2024-03-16T12:19:05.911444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858383+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:18:46.858383+0000","last_peered":"2024-03-16T12:18:46.858383+0000","last_clean":"2024-03-16T12:18:46.858383+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:18:46.858383+0000","last_undegraded":"2024-03-16T12:18:46.858383+0000","last_fullsized":"2024-03-16T12:18:46.858383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859184+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:18:46.859184+0000","last_peered":"2024-03-16T12:18:46.859184+0000","last_clean":"2024-03-16T12:18:46.859184+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:18:46.859184+0000","last_undegraded":"2024-03-16T12:18:46.859184+0000","last_fullsized":"2024-03-16T12:18:46.859184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858562+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:18:46.858562+0000","last_peered":"2024-03-16T12:18:46.858562+0000","last_clean":"2024-03-16T12:18:46.858562+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:18:46.858562+0000","last_undegraded":"2024-03-16T12:18:46.858562+0000","last_fullsized":"2024-03-16T12:18:46.858562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.352540+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:18:46.352540+0000","last_peered":"2024-03-16T12:18:46.352540+0000","last_clean":"2024-03-16T12:18:46.352540+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:18:46.352540+0000","last_undegraded":"2024-03-16T12:18:46.352540+0000","last_fullsized":"2024-03-16T12:18:46.352540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858791+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:18:46.858791+0000","last_peered":"2024-03-16T12:18:46.858791+0000","last_clean":"2024-03-16T12:18:46.858791+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:18:46.858791+0000","last_undegraded":"2024-03-16T12:18:46.858791+0000","last_fullsized":"2024-03-16T12:18:46.858791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:19:05.907575+0000","last_change":"2024-03-16T12:18:39.981656+0000","last_active":"2024-03-16T12:19:05.907575+0000","last_peered":"2024-03-16T12:19:05.907575+0000","last_clean":"2024-03-16T12:19:05.907575+0000","last_became_active":"2024-03-16T12:18:39.981300+0000","last_became_peered":"2024-03-16T12:18:39.981300+0000","last_unstale":"2024-03-16T12:19:05.907575+0000","last_undegraded":"2024-03-16T12:19:05.907575+0000","last_fullsized":"2024-03-16T12:19:05.907575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:09.167126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556204+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:18:45.556204+0000","last_peered":"2024-03-16T12:18:45.556204+0000","last_clean":"2024-03-16T12:18:45.556204+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:18:45.556204+0000","last_undegraded":"2024-03-16T12:18:45.556204+0000","last_fullsized":"2024-03-16T12:18:45.556204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858700+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:18:46.858700+0000","last_peered":"2024-03-16T12:18:46.858700+0000","last_clean":"2024-03-16T12:18:46.858700+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:18:46.858700+0000","last_undegraded":"2024-03-16T12:18:46.858700+0000","last_fullsized":"2024-03-16T12:18:46.858700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859423+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:18:46.859423+0000","last_peered":"2024-03-16T12:18:46.859423+0000","last_clean":"2024-03-16T12:18:46.859423+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:18:46.859423+0000","last_undegraded":"2024-03-16T12:18:46.859423+0000","last_fullsized":"2024-03-16T12:18:46.859423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556602+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:18:45.556602+0000","last_peered":"2024-03-16T12:18:45.556602+0000","last_clean":"2024-03-16T12:18:45.556602+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:18:45.556602+0000","last_undegraded":"2024-03-16T12:18:45.556602+0000","last_fullsized":"2024-03-16T12:18:45.556602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556512+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:18:45.556512+0000","last_peered":"2024-03-16T12:18:45.556512+0000","last_clean":"2024-03-16T12:18:45.556512+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:18:45.556512+0000","last_undegraded":"2024-03-16T12:18:45.556512+0000","last_fullsized":"2024-03-16T12:18:45.556512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.219302+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:18:46.219302+0000","last_peered":"2024-03-16T12:18:46.219302+0000","last_clean":"2024-03-16T12:18:46.219302+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:18:46.219302+0000","last_undegraded":"2024-03-16T12:18:46.219302+0000","last_fullsized":"2024-03-16T12:18:46.219302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556345+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:18:45.556345+0000","last_peered":"2024-03-16T12:18:45.556345+0000","last_clean":"2024-03-16T12:18:45.556345+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:18:45.556345+0000","last_undegraded":"2024-03-16T12:18:45.556345+0000","last_fullsized":"2024-03-16T12:18:45.556345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218571+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:18:46.218571+0000","last_peered":"2024-03-16T12:18:46.218571+0000","last_clean":"2024-03-16T12:18:46.218571+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:18:46.218571+0000","last_undegraded":"2024-03-16T12:18:46.218571+0000","last_fullsized":"2024-03-16T12:18:46.218571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.561255+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:18:45.561255+0000","last_peered":"2024-03-16T12:18:45.561255+0000","last_clean":"2024-03-16T12:18:45.561255+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:18:45.561255+0000","last_undegraded":"2024-03-16T12:18:45.561255+0000","last_fullsized":"2024-03-16T12:18:45.561255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556522+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:18:45.556522+0000","last_peered":"2024-03-16T12:18:45.556522+0000","last_clean":"2024-03-16T12:18:45.556522+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:18:45.556522+0000","last_undegraded":"2024-03-16T12:18:45.556522+0000","last_fullsized":"2024-03-16T12:18:45.556522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858509+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:18:46.858509+0000","last_peered":"2024-03-16T12:18:46.858509+0000","last_clean":"2024-03-16T12:18:46.858509+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:18:46.858509+0000","last_undegraded":"2024-03-16T12:18:46.858509+0000","last_fullsized":"2024-03-16T12:18:46.858509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859648+0000","last_change":"2024-03-16T12:18:40.501669+0000","last_active":"2024-03-16T12:18:46.859648+0000","last_peered":"2024-03-16T12:18:46.859648+0000","last_clean":"2024-03-16T12:18:46.859648+0000","last_became_active":"2024-03-16T12:18:40.499792+0000","last_became_peered":"2024-03-16T12:18:40.499792+0000","last_unstale":"2024-03-16T12:18:46.859648+0000","last_undegraded":"2024-03-16T12:18:46.859648+0000","last_fullsized":"2024-03-16T12:18:46.859648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:36.535106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858449+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:18:46.858449+0000","last_peered":"2024-03-16T12:18:46.858449+0000","last_clean":"2024-03-16T12:18:46.858449+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:18:46.858449+0000","last_undegraded":"2024-03-16T12:18:46.858449+0000","last_fullsized":"2024-03-16T12:18:46.858449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.556684+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:18:45.556684+0000","last_peered":"2024-03-16T12:18:45.556684+0000","last_clean":"2024-03-16T12:18:45.556684+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:18:45.556684+0000","last_undegraded":"2024-03-16T12:18:45.556684+0000","last_fullsized":"2024-03-16T12:18:45.556684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559470+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:18:45.559470+0000","last_peered":"2024-03-16T12:18:45.559470+0000","last_clean":"2024-03-16T12:18:45.559470+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:18:45.559470+0000","last_undegraded":"2024-03-16T12:18:45.559470+0000","last_fullsized":"2024-03-16T12:18:45.559470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858801+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:18:46.858801+0000","last_peered":"2024-03-16T12:18:46.858801+0000","last_clean":"2024-03-16T12:18:46.858801+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:18:46.858801+0000","last_undegraded":"2024-03-16T12:18:46.858801+0000","last_fullsized":"2024-03-16T12:18:46.858801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859265+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:18:46.859265+0000","last_peered":"2024-03-16T12:18:46.859265+0000","last_clean":"2024-03-16T12:18:46.859265+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:18:46.859265+0000","last_undegraded":"2024-03-16T12:18:46.859265+0000","last_fullsized":"2024-03-16T12:18:46.859265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353616+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:18:46.353616+0000","last_peered":"2024-03-16T12:18:46.353616+0000","last_clean":"2024-03-16T12:18:46.353616+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:18:46.353616+0000","last_undegraded":"2024-03-16T12:18:46.353616+0000","last_fullsized":"2024-03-16T12:18:46.353616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.218827+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:18:46.218827+0000","last_peered":"2024-03-16T12:18:46.218827+0000","last_clean":"2024-03-16T12:18:46.218827+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:18:46.218827+0000","last_undegraded":"2024-03-16T12:18:46.218827+0000","last_fullsized":"2024-03-16T12:18:46.218827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.353267+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:18:46.353267+0000","last_peered":"2024-03-16T12:18:46.353267+0000","last_clean":"2024-03-16T12:18:46.353267+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:18:46.353267+0000","last_undegraded":"2024-03-16T12:18:46.353267+0000","last_fullsized":"2024-03-16T12:18:46.353267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.858974+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:18:46.858974+0000","last_peered":"2024-03-16T12:18:46.858974+0000","last_clean":"2024-03-16T12:18:46.858974+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:18:46.858974+0000","last_undegraded":"2024-03-16T12:18:46.858974+0000","last_fullsized":"2024-03-16T12:18:46.858974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:45.559306+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:18:45.559306+0000","last_peered":"2024-03-16T12:18:45.559306+0000","last_clean":"2024-03-16T12:18:45.559306+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:18:45.559306+0000","last_undegraded":"2024-03-16T12:18:45.559306+0000","last_fullsized":"2024-03-16T12:18:45.559306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-16T12:18:46.859293+0000","last_change":"2024-03-16T12:18:40.502717+0000","last_active":"2024-03-16T12:18:46.859293+0000","last_peered":"2024-03-16T12:18:46.859293+0000","last_clean":"2024-03-16T12:18:46.859293+0000","last_became_active":"2024-03-16T12:18:40.502372+0000","last_became_peered":"2024-03-16T12:18:40.502372+0000","last_unstale":"2024-03-16T12:18:46.859293+0000","last_undegraded":"2024-03-16T12:18:46.859293+0000","last_fullsized":"2024-03-16T12:18:46.859293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:31:23.058685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6870,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":22,"num_read_kb":18,"num_write":121,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":18940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1597440,"data_stored":1073471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495662,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1609728,"data_stored":1073155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:19:11.871 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T12:19:11.871 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T12:19:11.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T12:19:12.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:11 smithi098 ceph-mon[31633]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-16T12:19:12.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:11 smithi153 ceph-mon[36901]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-16T12:19:12.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:11 smithi173 ceph-mon[36632]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-16T12:19:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:13.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:12 smithi098 ceph-mon[31633]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:13.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:12 smithi098 ceph-mon[31633]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:13.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:12 smithi153 ceph-mon[36901]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:13.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:12 smithi153 ceph-mon[36901]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:13.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:12 smithi173 ceph-mon[36632]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:13.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:12 smithi173 ceph-mon[36632]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:19:14.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:13 smithi098 ceph-mon[31633]: pgmap v334: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-16T12:19:14.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:13 smithi153 ceph-mon[36901]: pgmap v334: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-16T12:19:14.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:13 smithi173 ceph-mon[36632]: pgmap v334: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-16T12:19:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 -- 172.21.15.98:0/4170190402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0102340 msgr2=0x7f93c0102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 --2- 172.21.15.98:0/4170190402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0102340 0x7f93c0102760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009990 tx=0x7f93b0009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 -- 172.21.15.98:0/4170190402 shutdown_connections 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 --2- 172.21.15.98:0/4170190402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0103ee0 0x7f93c010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 --2- 172.21.15.98:0/4170190402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0102340 0x7f93c0102760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 --2- 172.21.15.98:0/4170190402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 -- 172.21.15.98:0/4170190402 >> 172.21.15.98:0/4170190402 conn(0x7f93c00fd8e0 msgr2=0x7f93c00ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 -- 172.21.15.98:0/4170190402 shutdown_connections 2024-03-16T12:19:14.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.736+0000 7f93c6400700 1 -- 172.21.15.98:0/4170190402 wait complete. 2024-03-16T12:19:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.737+0000 7f93c6400700 1 Processor -- start 2024-03-16T12:19:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 -- start start 2024-03-16T12:19:14.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0102340 0x7f93c019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 0x7f93c01a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010df70 con 0x7f93c0102340 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010ddf0 con 0x7f93c0103540 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93c6400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010e0f0 con 0x7f93c0103ee0 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93bea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93bea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:47930/0 (socket says 172.21.15.98:47930) 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93bea00700 1 -- 172.21.15.98:0/2112619912 learned_addr learned my addr 172.21.15.98:0/2112619912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93bf400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0102340 0x7f93c019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.738+0000 7f93bfe00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 0x7f93c01a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 -- 172.21.15.98:0/2112619912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 msgr2=0x7f93c01a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:14.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 0x7f93c01a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 -- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0102340 msgr2=0x7f93c019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0102340 0x7f93c019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 -- 172.21.15.98:0/2112619912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c01a23c0 con 0x7f93c0103540 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bfe00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 0x7f93c01a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93bea00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00e9c0 tx=0x7f93ac00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:14.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00cbd0 con 0x7f93c0103540 2024-03-16T12:19:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.739+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c01a26b0 con 0x7f93c0103540 2024-03-16T12:19:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.740+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93ac00ee60 con 0x7f93c0103540 2024-03-16T12:19:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.740+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac005680 con 0x7f93c0103540 2024-03-16T12:19:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.740+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c01a2bf0 con 0x7f93c0103540 2024-03-16T12:19:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.741+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c00507e0 con 0x7f93c0103540 2024-03-16T12:19:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.743+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f93ac004750 con 0x7f93c0103540 2024-03-16T12:19:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.744+0000 7f93b7e00700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 0x7f93a40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.744+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93ac0cf0c0 con 0x7f93c0103540 2024-03-16T12:19:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.744+0000 7f93bf400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 0x7f93a40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:14.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.745+0000 7f93bf400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 0x7f93a40a4e90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93b000b820 tx=0x7f93b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:14.752+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93ac09bd60 con 0x7f93c0103540 2024-03-16T12:19:15.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.069+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f93c01a3040 con 0x7f93c0103540 2024-03-16T12:19:15.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.076+0000 7f93b7e00700 1 -- 172.21.15.98:0/2112619912 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f93ac09b660 con 0x7f93c0103540 2024-03-16T12:19:15.058 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-16T12:19:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBDjvVl20FPBBAAQ+P0tX6iqJ+lX+SVKBJQ3g== 2024-03-16T12:19:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.079+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 msgr2=0x7f93a40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.079+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 0x7f93a40a4e90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93b000b820 tx=0x7f93b0005fd0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.079+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 msgr2=0x7f93c019aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.079+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00e9c0 tx=0x7f93ac00ecd0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 shutdown_connections 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c0103ee0 0x7f93c01a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93c0102340 0x7f93c019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f93a40a29d0 0x7f93a40a4e90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 --2- 172.21.15.98:0/2112619912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93c0103540 0x7f93c019aca0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 >> 172.21.15.98:0/2112619912 conn(0x7f93c00fd8e0 msgr2=0x7f93c0074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 shutdown_connections 2024-03-16T12:19:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:15.080+0000 7f93c6400700 1 -- 172.21.15.98:0/2112619912 wait complete. 2024-03-16T12:19:16.084 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:19:16.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T12:19:16.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T12:19:16.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:15 smithi098 ceph-mon[31633]: pgmap v335: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-16T12:19:16.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:15 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2112619912' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:15 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:15 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T12:19:16.236 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T12:19:16.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T12:19:16.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T12:19:16.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:15 smithi153 ceph-mon[36901]: pgmap v335: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-16T12:19:16.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:15 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2112619912' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:15 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:15 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T12:19:16.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:15 smithi173 ceph-mon[36632]: pgmap v335: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-16T12:19:16.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:15 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2112619912' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:15 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T12:19:16.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:15 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T12:19:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:17 smithi098 ceph-mon[31633]: pgmap v336: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-16T12:19:18.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:17 smithi153 ceph-mon[36901]: pgmap v336: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-16T12:19:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:17 smithi173 ceph-mon[36632]: pgmap v336: 129 pgs: 129 active+clean; 620 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-16T12:19:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 -- 172.21.15.98:0/934027098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 msgr2=0x7ff5d810e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:18.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 --2- 172.21.15.98:0/934027098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d810e840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40098f0 tx=0x7ff5c4009c00 comp rx=0 tx=0).stop 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 -- 172.21.15.98:0/934027098 shutdown_connections 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 --2- 172.21.15.98:0/934027098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d806a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 --2- 172.21.15.98:0/934027098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 0x7ff5d8111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 --2- 172.21.15.98:0/934027098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d810e840 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.874+0000 7ff5e0000700 1 -- 172.21.15.98:0/934027098 >> 172.21.15.98:0/934027098 conn(0x7ff5d8078100 msgr2=0x7ff5d8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.875+0000 7ff5e0000700 1 -- 172.21.15.98:0/934027098 shutdown_connections 2024-03-16T12:19:18.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.875+0000 7ff5e0000700 1 -- 172.21.15.98:0/934027098 wait complete. 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 Processor -- start 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 -- start start 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 0x7ff5d819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d8114810 con 0x7ff5d806a8a0 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d8114690 con 0x7ff5d8069f80 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.876+0000 7ff5e0000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d8114990 con 0x7ff5d810ed80 2024-03-16T12:19:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58428/0 (socket says 172.21.15.98:58428) 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 -- 172.21.15.98:0/3318195669 learned_addr learned my addr 172.21.15.98:0/3318195669 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dcc00700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5de000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 0x7ff5d819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 -- 172.21.15.98:0/3318195669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 msgr2=0x7ff5d819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 0x7ff5d819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 -- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 msgr2=0x7ff5d819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d819b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 -- 172.21.15.98:0/3318195669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d819fe20 con 0x7ff5d8069f80 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.877+0000 7ff5dd600700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00ef10 tx=0x7ff5cc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.878+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc00c9b0 con 0x7ff5d8069f80 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.878+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc019070 con 0x7ff5d8069f80 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.878+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d81a00b0 con 0x7ff5d8069f80 2024-03-16T12:19:18.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.878+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc007e60 con 0x7ff5d8069f80 2024-03-16T12:19:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.878+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d81a7990 con 0x7ff5d8069f80 2024-03-16T12:19:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.879+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d8068f90 con 0x7ff5d8069f80 2024-03-16T12:19:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.881+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff5cc012050 con 0x7ff5d8069f80 2024-03-16T12:19:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.882+0000 7ff5d6000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 0x7ff5c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.882+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff5cc0cee40 con 0x7ff5d8069f80 2024-03-16T12:19:18.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.882+0000 7ff5dcc00700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 0x7ff5c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:18.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.883+0000 7ff5dcc00700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 0x7ff5c00a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009810 tx=0x7ff5c4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:18.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:18.884+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5cc010040 con 0x7ff5d8069f80 2024-03-16T12:19:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.166+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff5d802db70 con 0x7ff5d8069f80 2024-03-16T12:19:19.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.171+0000 7ff5d6000700 1 -- 172.21.15.98:0/3318195669 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262659 (secure 0 0 0) 0x7ff5cc09bae0 con 0x7ff5d8069f80 2024-03-16T12:19:19.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 msgr2=0x7ff5c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 0x7ff5c00a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009810 tx=0x7ff5c4019040 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 msgr2=0x7ff5d819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00ef10 tx=0x7ff5cc00c3a0 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 shutdown_connections 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5c00a2920 0x7ff5c00a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5d8069f80 0x7ff5d819ae10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5d810ed80 0x7ff5d819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 --2- 172.21.15.98:0/3318195669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5d806a8a0 0x7ff5d819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 >> 172.21.15.98:0/3318195669 conn(0x7ff5d8078100 msgr2=0x7ff5d8110190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 shutdown_connections 2024-03-16T12:19:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:19.178+0000 7ff5e0000700 1 -- 172.21.15.98:0/3318195669 wait complete. 2024-03-16T12:19:20.038 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3089653509},{"type":"v1","addr":"172.21.15.98:6801","nonce":3089653509}]},"active_addr":"172.21.15.98:6801/3089653509","active_change":"2024-03-16T12:10:42.633257+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3627906864}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":538510969}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":4287586498}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2698332152}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":895186239}]}]} 2024-03-16T12:19:20.042 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T12:19:20.042 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T12:19:20.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T12:19:20.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:19 smithi098 ceph-mon[31633]: pgmap v337: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-03-16T12:19:20.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:19 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3318195669' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T12:19:20.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:19 smithi153 ceph-mon[36901]: pgmap v337: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-03-16T12:19:20.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:19 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3318195669' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T12:19:20.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:19 smithi173 ceph-mon[36632]: pgmap v337: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-03-16T12:19:20.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:19 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3318195669' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T12:19:20.535 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:21 smithi098 ceph-mon[31633]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-16T12:19:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:21 smithi153 ceph-mon[36901]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-16T12:19:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:21 smithi173 ceph-mon[36632]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-16T12:19:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 -- 172.21.15.98:0/265928046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc534104ea0 msgr2=0x7fc534111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 --2- 172.21.15.98:0/265928046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc534104ea0 0x7fc534111360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009f90 tx=0x7fc52c00b330 comp rx=0 tx=0).stop 2024-03-16T12:19:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 -- 172.21.15.98:0/265928046 shutdown_connections 2024-03-16T12:19:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 --2- 172.21.15.98:0/265928046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc53410a590 0x7fc53410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:22.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 --2- 172.21.15.98:0/265928046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc534104ea0 0x7fc534111360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:22.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 --2- 172.21.15.98:0/265928046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5341044e0 0x7fc534104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:22.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.879+0000 7fc53b800700 1 -- 172.21.15.98:0/265928046 >> 172.21.15.98:0/265928046 conn(0x7fc5340fffb0 msgr2=0x7fc5341023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:22.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.880+0000 7fc53b800700 1 -- 172.21.15.98:0/265928046 shutdown_connections 2024-03-16T12:19:22.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.880+0000 7fc53b800700 1 -- 172.21.15.98:0/265928046 wait complete. 2024-03-16T12:19:22.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.881+0000 7fc53b800700 1 Processor -- start 2024-03-16T12:19:22.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.881+0000 7fc53b800700 1 -- start start 2024-03-16T12:19:22.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:22.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc534104ea0 0x7fc5341a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc53410a590 0x7fc53419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc534114840 con 0x7fc53410a590 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5341146c0 con 0x7fc534104ea0 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc53b800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc5341149c0 con 0x7fc5341044e0 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc533e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc534104ea0 0x7fc5341a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:53978/0 (socket says 172.21.15.98:53978) 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 -- 172.21.15.98:0/2315154282 learned_addr learned my addr 172.21.15.98:0/2315154282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc539800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc53410a590 0x7fc53419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 -- 172.21.15.98:0/2315154282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc534104ea0 msgr2=0x7fc5341a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc534104ea0 0x7fc5341a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:22.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 -- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc53410a590 msgr2=0x7fc53419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc53410a590 0x7fc53419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.882+0000 7fc538e00700 1 -- 172.21.15.98:0/2315154282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53419b0a0 con 0x7fc5341044e0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc538e00700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e9f0 tx=0x7fc52800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528004a30 con 0x7fc5341044e0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc52800cca0 con 0x7fc5341044e0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53419b390 con 0x7fc5341044e0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280056c0 con 0x7fc5341044e0 2024-03-16T12:19:22.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.883+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5341a7650 con 0x7fc5341044e0 2024-03-16T12:19:22.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.884+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc534068f90 con 0x7fc5341044e0 2024-03-16T12:19:22.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.886+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc52800c7c0 con 0x7fc5341044e0 2024-03-16T12:19:22.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.888+0000 7fc531600700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 0x7fc51c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:22.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.888+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc5280cef90 con 0x7fc5341044e0 2024-03-16T12:19:22.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.888+0000 7fc533e00700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 0x7fc51c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:22.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.889+0000 7fc533e00700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 0x7fc51c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc534109a50 tx=0x7fc520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:22.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:22.891+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc52809bc30 con 0x7fc5341044e0 2024-03-16T12:19:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.118+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5341a78e0 con 0x7fc5341044e0 2024-03-16T12:19:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.120+0000 7fc531600700 1 -- 172.21.15.98:0/2315154282 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7fc52809b530 con 0x7fc5341044e0 2024-03-16T12:19:23.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:23.102 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 msgr2=0x7fc51c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 0x7fc51c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc534109a50 tx=0x7fc520009260 comp rx=0 tx=0).stop 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 msgr2=0x7fc5341a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e9f0 tx=0x7fc52800ed00 comp rx=0 tx=0).stop 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 shutdown_connections 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc53410a590 0x7fc53419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5341044e0 0x7fc5341a0780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc51c0a29a0 0x7fc51c0a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 --2- 172.21.15.98:0/2315154282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc534104ea0 0x7fc5341a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.123+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 >> 172.21.15.98:0/2315154282 conn(0x7fc5340fffb0 msgr2=0x7fc534101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.124+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 shutdown_connections 2024-03-16T12:19:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:23.124+0000 7fc53b800700 1 -- 172.21.15.98:0/2315154282 wait complete. 2024-03-16T12:19:23.914 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T12:19:23.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T12:19:24.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:23 smithi098 ceph-mon[31633]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-03-16T12:19:24.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2315154282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:24.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:23 smithi153 ceph-mon[36901]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-03-16T12:19:24.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2315154282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:23 smithi173 ceph-mon[36632]: pgmap v339: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-03-16T12:19:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2315154282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:24.364 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:26.112 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:25 smithi098 ceph-mon[31633]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T12:19:26.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:25 smithi153 ceph-mon[36901]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T12:19:26.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:25 smithi173 ceph-mon[36632]: pgmap v340: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 -- 172.21.15.98:0/2807856453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 msgr2=0x7f38d0108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 --2- 172.21.15.98:0/2807856453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0108700 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f38c00099c0 tx=0x7f38c0009cd0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 -- 172.21.15.98:0/2807856453 shutdown_connections 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 --2- 172.21.15.98:0/2807856453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d006a920 0x7f38d010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 --2- 172.21.15.98:0/2807856453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0108700 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 --2- 172.21.15.98:0/2807856453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d0069f80 0x7f38d006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 -- 172.21.15.98:0/2807856453 >> 172.21.15.98:0/2807856453 conn(0x7f38d0078100 msgr2=0x7f38d0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 -- 172.21.15.98:0/2807856453 shutdown_connections 2024-03-16T12:19:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.747+0000 7f38d6200700 1 -- 172.21.15.98:0/2807856453 wait complete. 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.748+0000 7f38d6200700 1 Processor -- start 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.748+0000 7f38d6200700 1 -- start start 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d0069f80 0x7f38d01012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 0x7f38d01017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38d01125b0 con 0x7f38d0069f80 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0112430 con 0x7f38d006a920 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38d6200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0112730 con 0x7f38d0108320 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38c6a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 0x7f38d01017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38cf400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d0069f80 0x7f38d01012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38cfe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38c6a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 0x7f38d01017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58464/0 (socket says 172.21.15.98:58464) 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.749+0000 7f38c6a00700 1 -- 172.21.15.98:0/2127010768 learned_addr learned my addr 172.21.15.98:0/2127010768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 -- 172.21.15.98:0/2127010768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 msgr2=0x7f38d01017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 0x7f38d01017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 -- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d0069f80 msgr2=0x7f38d01012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d0069f80 0x7f38d01012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 -- 172.21.15.98:0/2127010768 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d01a6f90 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38cfe00700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0101d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38c800efc0 tx=0x7f38c800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c80092e0 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c8004bb0 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8004280 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d01a7220 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.750+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d01a7710 con 0x7f38d0108320 2024-03-16T12:19:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.751+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d0068f90 con 0x7f38d0108320 2024-03-16T12:19:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.753+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f38c801b030 con 0x7f38d0108320 2024-03-16T12:19:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.754+0000 7f38ccc00700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 0x7f38b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.755+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38c80ce960 con 0x7f38d0108320 2024-03-16T12:19:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.755+0000 7f38cf400700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 0x7f38b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.755+0000 7f38cf400700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 0x7f38b00a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f38c0000c00 tx=0x7f38c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.759+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38c8022020 con 0x7f38d0108320 2024-03-16T12:19:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.987+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38d002dd30 con 0x7f38d0108320 2024-03-16T12:19:26.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.990+0000 7f38ccc00700 1 -- 172.21.15.98:0/2127010768 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24126 (secure 0 0 0) 0x7f38c8010040 con 0x7f38d0108320 2024-03-16T12:19:26.972 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:19:26.973 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:18:45.024195+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 msgr2=0x7f38b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 0x7f38b00a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f38c0000c00 tx=0x7f38c0005fb0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 msgr2=0x7f38d0101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0101d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38c800efc0 tx=0x7f38c800c3a0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 shutdown_connections 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d0108320 0x7f38d0101d20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d0069f80 0x7f38d01012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38b00a2920 0x7f38b00a4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 --2- 172.21.15.98:0/2127010768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d006a920 0x7f38d01017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 >> 172.21.15.98:0/2127010768 conn(0x7f38d0078100 msgr2=0x7f38d00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:26.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 shutdown_connections 2024-03-16T12:19:26.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:26.994+0000 7f38d6200700 1 -- 172.21.15.98:0/2127010768 wait complete. 2024-03-16T12:19:27.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:26 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2127010768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:27.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:26 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2127010768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:27.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:26 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2127010768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:19:27.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T12:19:27.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T12:19:27.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T12:19:27.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T12:19:27.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T12:19:27.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T12:19:27.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T12:19:27.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T12:19:27.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T12:19:27.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T12:19:27.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T12:19:27.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T12:19:27.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:28 smithi098 ceph-mon[31633]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T12:19:28.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:28 smithi153 ceph-mon[36901]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T12:19:28.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:28 smithi173 ceph-mon[36632]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T12:19:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:28.947 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:28.970 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:28.982 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:30.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:30 smithi098 ceph-mon[31633]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T12:19:30.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:30 smithi153 ceph-mon[36901]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T12:19:30.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:30 smithi173 ceph-mon[36632]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T12:19:30.871 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:32.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:32 smithi153 ceph-mon[36901]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:32.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:32 smithi173 ceph-mon[36632]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:32.376 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:32 smithi098 ceph-mon[31633]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:32.396 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:32.413 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:33.717 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:33 smithi098 ceph-mon[31633]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:33.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:33 smithi153 ceph-mon[36901]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:33.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:33 smithi173 ceph-mon[36632]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T12:19:35.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:35 smithi098 ceph-mon[31633]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:19:36.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:35 smithi153 ceph-mon[36901]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:19:36.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:35 smithi173 ceph-mon[36632]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:19:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 -- 172.21.15.98:0/1577431126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6141088e0 msgr2=0x7fe614108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 --2- 172.21.15.98:0/1577431126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6141088e0 0x7fe614108d60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe600009990 tx=0x7fe600009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 -- 172.21.15.98:0/1577431126 shutdown_connections 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 --2- 172.21.15.98:0/1577431126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe6141157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 --2- 172.21.15.98:0/1577431126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe61410e8e0 0x7fe61410ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 --2- 172.21.15.98:0/1577431126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6141088e0 0x7fe614108d60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 -- 172.21.15.98:0/1577431126 >> 172.21.15.98:0/1577431126 conn(0x7fe614075aa0 msgr2=0x7fe614077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 -- 172.21.15.98:0/1577431126 shutdown_connections 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.060+0000 7fe61b000700 1 -- 172.21.15.98:0/1577431126 wait complete. 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 Processor -- start 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 -- start start 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 0x7fe61419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe61419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe61410e8e0 0x7fe6141a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe614118cc0 con 0x7fe6141088e0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe614118b40 con 0x7fe6141092a0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe61b000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe614118e40 con 0x7fe61410e8e0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.061+0000 7fe613400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe61419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 0x7fe61419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 0x7fe61419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51126/0 (socket says 172.21.15.98:51126) 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe61419f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:34378/0 (socket says 172.21.15.98:34378) 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 -- 172.21.15.98:0/4010692487 learned_addr learned my addr 172.21.15.98:0/4010692487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe619000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe61410e8e0 0x7fe6141a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 -- 172.21.15.98:0/4010692487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe61410e8e0 msgr2=0x7fe6141a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe61410e8e0 0x7fe6141a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 -- 172.21.15.98:0/4010692487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 msgr2=0x7fe61419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe61419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6141a4080 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613400700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6141092a0 0x7fe61419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.062+0000 7fe613e00700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 0x7fe61419f070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe60400e9a0 tx=0x7fe60400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60400ca10 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6141a4370 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe60400cbb0 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604018680 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6141abbf0 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.063+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe614062a60 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.066+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe604010430 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.067+0000 7fe610c00700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe5f80a2920 0x7fe5f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.067+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe6040cf7f0 con 0x7fe6141088e0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.067+0000 7fe613400700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe5f80a2920 0x7fe5f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe610c00700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] conn(0x7fe5f80ad5e0 0x7fe5f80afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe60401bec0 con 0x7fe5f80ad5e0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe6040cfba0 con 0x7fe6141088e0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe619000700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] conn(0x7fe5f80ad5e0 0x7fe5f80afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe613400700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe5f80a2920 0x7fe5f80a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe614078010 tx=0x7fe60000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.068+0000 7fe619000700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] conn(0x7fe5f80ad5e0 0x7fe5f80afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.069+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== osd.6 v2:172.21.15.98:6818/3005930753 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe60401bec0 con 0x7fe5f80ad5e0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.087+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 --> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe614062e60 con 0x7fe5f80ad5e0 2024-03-16T12:19:36.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe610c00700 1 -- 172.21.15.98:0/4010692487 <== osd.6 v2:172.21.15.98:6818/3005930753 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe614062e60 con 0x7fe5f80ad5e0 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] conn(0x7fe5f80ad5e0 msgr2=0x7fe5f80afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] conn(0x7fe5f80ad5e0 0x7fe5f80afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe5f80a2920 msgr2=0x7fe5f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe5f80a2920 0x7fe5f80a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe614078010 tx=0x7fe60000b260 comp rx=0 tx=0).stop 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 msgr2=0x7fe61419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 --2- 172.21.15.98:0/4010692487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6141088e0 0x7fe61419f070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe60400e9a0 tx=0x7fe60400ecb0 comp rx=0 tx=0).stop 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe619000700 1 -- 172.21.15.98:0/4010692487 reap_dead start 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.088+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 shutdown_connections 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.089+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 >> 172.21.15.98:0/4010692487 conn(0x7fe614075aa0 msgr2=0x7fe614076050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.089+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 shutdown_connections 2024-03-16T12:19:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:34.089+0000 7fe61b000700 1 -- 172.21.15.98:0/4010692487 wait complete. 2024-03-16T12:19:38.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:37 smithi098 ceph-mon[31633]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:19:38.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:37 smithi153 ceph-mon[36901]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:19:38.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:37 smithi173 ceph-mon[36632]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:19:40.062 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:39 smithi098 ceph-mon[31633]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 596 B/s wr, 6 op/s 2024-03-16T12:19:40.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:39 smithi153 ceph-mon[36901]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 596 B/s wr, 6 op/s 2024-03-16T12:19:40.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:39 smithi173 ceph-mon[36632]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 596 B/s wr, 6 op/s 2024-03-16T12:19:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 -- 172.21.15.98:0/960622634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 msgr2=0x7f25a410e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 --2- 172.21.15.98:0/960622634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a410e8a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f258c009990 tx=0x7f258c009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 -- 172.21.15.98:0/960622634 shutdown_connections 2024-03-16T12:19:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 --2- 172.21.15.98:0/960622634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a410ede0 0x7f25a41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 --2- 172.21.15.98:0/960622634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a406a9d0 0x7f25a406adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 --2- 172.21.15.98:0/960622634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a410e8a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 -- 172.21.15.98:0/960622634 >> 172.21.15.98:0/960622634 conn(0x7f25a4077220 msgr2=0x7f25a4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 -- 172.21.15.98:0/960622634 shutdown_connections 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.411+0000 7f25aaa00700 1 -- 172.21.15.98:0/960622634 wait complete. 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.412+0000 7f25aaa00700 1 Processor -- start 2024-03-16T12:19:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.412+0000 7f25aaa00700 1 -- start start 2024-03-16T12:19:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a406a9d0 0x7f25a419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a410ede0 0x7f25a419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4114840 con 0x7f25a410ede0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41146c0 con 0x7f25a406a9d0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25aaa00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41149c0 con 0x7f25a406b2f0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25a3e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a406a9d0 0x7f25a419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f259b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25a8a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a410ede0 0x7f25a419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f259b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a419b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52776/0 (socket says 172.21.15.98:52776) 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f259b400700 1 -- 172.21.15.98:0/2824710353 learned_addr learned my addr 172.21.15.98:0/2824710353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.413+0000 7f25a3e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a406a9d0 0x7f25a419ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:42750/0 (socket says 172.21.15.98:42750) 2024-03-16T12:19:40.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 -- 172.21.15.98:0/2824710353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a406a9d0 msgr2=0x7f25a419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25a406a9d0 0x7f25a419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 -- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a410ede0 msgr2=0x7f25a419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25a410ede0 0x7f25a419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a419fd20 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f259b400700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a419b250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f258c000c00 tx=0x7f258c00bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c027070 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f258c0056d0 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c022e10 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a419ffb0 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.414+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a7890 con 0x7f25a406b2f0 2024-03-16T12:19:40.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.415+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25a40629e0 con 0x7f25a406b2f0 2024-03-16T12:19:40.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.417+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f258c02a070 con 0x7f25a406b2f0 2024-03-16T12:19:40.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.418+0000 7f25a1600700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f25880a29a0 0x7f25880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:40.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f258c0cfe00 con 0x7f25a406b2f0 2024-03-16T12:19:40.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a3e00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f25880a29a0 0x7f25880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:40.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a1600700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] conn(0x7f25880ad4b0 0x7f25880af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:40.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25880affc0 con 0x7f25880ad4b0 2024-03-16T12:19:40.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f258c0d01b0 con 0x7f25a406b2f0 2024-03-16T12:19:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.419+0000 7f25a8a00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] conn(0x7f25880ad4b0 0x7f25880af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.420+0000 7f25a3e00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f25880a29a0 0x7f25880a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2594009c60 tx=0x7f2594009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.420+0000 7f25a8a00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] conn(0x7f25880ad4b0 0x7f25880af8f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:40.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.421+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== osd.5 v2:172.21.15.173:6808/3808045803 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f25880affc0 con 0x7f25880ad4b0 2024-03-16T12:19:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.439+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 --> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f25a4062de0 con 0x7f25880ad4b0 2024-03-16T12:19:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.439+0000 7f25a1600700 1 -- 172.21.15.98:0/2824710353 <== osd.5 v2:172.21.15.173:6808/3808045803 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f25a4062de0 con 0x7f25880ad4b0 2024-03-16T12:19:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] conn(0x7f25880ad4b0 msgr2=0x7f25880af8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] conn(0x7f25880ad4b0 0x7f25880af8f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f25880a29a0 msgr2=0x7f25880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f25880a29a0 0x7f25880a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2594009c60 tx=0x7f2594009260 comp rx=0 tx=0).stop 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 msgr2=0x7f25a419b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 --2- 172.21.15.98:0/2824710353 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a406b2f0 0x7f25a419b250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f258c000c00 tx=0x7f258c00bab0 comp rx=0 tx=0).stop 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25a8a00700 1 -- 172.21.15.98:0/2824710353 reap_dead start 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 shutdown_connections 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 >> 172.21.15.98:0/2824710353 conn(0x7f25a4077220 msgr2=0x7f25a4110290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 shutdown_connections 2024-03-16T12:19:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.440+0000 7f25aaa00700 1 -- 172.21.15.98:0/2824710353 wait complete. 2024-03-16T12:19:40.843 INFO:teuthology.orchestra.run.smithi098.stdout:197568495668 2024-03-16T12:19:40.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 -- 172.21.15.98:0/1126674152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c102550 msgr2=0x7f2c6c102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 --2- 172.21.15.98:0/1126674152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c102550 0x7f2c6c102930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0099c0 tx=0x7f2c5c009cd0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 -- 172.21.15.98:0/1126674152 shutdown_connections 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 --2- 172.21.15.98:0/1126674152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c107630 0x7f2c6c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 --2- 172.21.15.98:0/1126674152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c102550 0x7f2c6c102930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 --2- 172.21.15.98:0/1126674152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 -- 172.21.15.98:0/1126674152 >> 172.21.15.98:0/1126674152 conn(0x7f2c6c0fdda0 msgr2=0x7f2c6c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 -- 172.21.15.98:0/1126674152 shutdown_connections 2024-03-16T12:19:41.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.248+0000 7f2c71400700 1 -- 172.21.15.98:0/1126674152 wait complete. 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.249+0000 7f2c71400700 1 Processor -- start 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.249+0000 7f2c71400700 1 -- start start 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c102550 0x7f2c6c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c107630 0x7f2c6c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c112660 con 0x7f2c6c102f00 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1124e0 con 0x7f2c6c102550 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c71400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1127e0 con 0x7f2c6c107630 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c69e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c102550 0x7f2c6c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c69e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c19ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55278/0 (socket says 172.21.15.98:55278) 2024-03-16T12:19:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c69e00700 1 -- 172.21.15.98:0/208614952 learned_addr learned my addr 172.21.15.98:0/208614952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6b200700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c107630 0x7f2c6c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 -- 172.21.15.98:0/208614952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c107630 msgr2=0x7f2c6c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c6c107630 0x7f2c6c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 -- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 msgr2=0x7f2c6c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.250+0000 7f2c6a800700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c199060 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c69e00700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c6c102f00 0x7f2c6c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c6a800700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c102550 0x7f2c6c19e740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098e0 tx=0x7f2c5c003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c003680 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c5c0055a0 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00fce0 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c1992f0 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.251+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c1a5610 con 0x7f2c6c102550 2024-03-16T12:19:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.252+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2c6c062a60 con 0x7f2c6c102550 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.254+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2c5c00f460 con 0x7f2c6c102550 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.255+0000 7f2c63400700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2c500a2b40 0x7f2c500a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2c5c0cffa0 con 0x7f2c6c102550 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c69e00700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2c500a2b40 0x7f2c500a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c63400700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] conn(0x7f2c500ad6d0 0x7f2c500afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c5c021e80 con 0x7f2c500ad6d0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2c5c0d0350 con 0x7f2c6c102550 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.256+0000 7f2c6b200700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] conn(0x7f2c500ad6d0 0x7f2c500afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.257+0000 7f2c69e00700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2c500a2b40 0x7f2c500a5000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ff2a0 tx=0x7f2c58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.257+0000 7f2c6b200700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] conn(0x7f2c500ad6d0 0x7f2c500afb10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.258+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== osd.0 v2:172.21.15.98:6802/3304246268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c5c021e80 con 0x7f2c500ad6d0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.295+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 --> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2c6c062e60 con 0x7f2c500ad6d0 2024-03-16T12:19:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.296+0000 7f2c63400700 1 -- 172.21.15.98:0/208614952 <== osd.0 v2:172.21.15.98:6802/3304246268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2c6c062e60 con 0x7f2c500ad6d0 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.296+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] conn(0x7f2c500ad6d0 msgr2=0x7f2c500afb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.296+0000 7f2c71400700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] conn(0x7f2c500ad6d0 0x7f2c500afb10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.296+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2c500a2b40 msgr2=0x7f2c500a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2c500a2b40 0x7f2c500a5000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ff2a0 tx=0x7f2c58009260 comp rx=0 tx=0).stop 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c102550 msgr2=0x7f2c6c19e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 --2- 172.21.15.98:0/208614952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c6c102550 0x7f2c6c19e740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098e0 tx=0x7f2c5c003a90 comp rx=0 tx=0).stop 2024-03-16T12:19:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c6b200700 1 -- 172.21.15.98:0/208614952 reap_dead start 2024-03-16T12:19:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 shutdown_connections 2024-03-16T12:19:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 >> 172.21.15.98:0/208614952 conn(0x7f2c6c0fdda0 msgr2=0x7f2c6c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 shutdown_connections 2024-03-16T12:19:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:40.297+0000 7f2c71400700 1 -- 172.21.15.98:0/208614952 wait complete. 2024-03-16T12:19:41.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:40 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:19:41.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:40 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:19:41.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:40 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:19:41.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:41 smithi098 ceph-mon[31633]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:42.070 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:41 smithi173 ceph-mon[36632]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:42.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:41 smithi153 ceph-mon[36901]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-16T12:19:43.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2024-03-16T12:19:43.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-16T12:19:43.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-16T12:19:43.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-16T12:19:43.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:43 smithi098 ceph-mon[31633]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:43.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T12:19:43.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:43.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:43 smithi173 ceph-mon[36632]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:44.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:44.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:44.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T12:19:44.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-16T12:19:44.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:19:44.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:43 smithi153 ceph-mon[36901]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-16T12:19:44.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:44 smithi098 ceph-mon[31633]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T12:19:44.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T12:19:44.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-16T12:19:44.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-16T12:19:44.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-16T12:19:44.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-16T12:19:44.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:44 smithi173 ceph-mon[36632]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T12:19:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-16T12:19:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [5, 10]}]': finished 2024-03-16T12:19:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-16T12:19:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-16T12:19:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-16T12:19:45.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:44 smithi153 ceph-mon[36901]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T12:19:45.295 INFO:teuthology.orchestra.run.smithi098.stdout:34359738455 2024-03-16T12:19:45.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T12:19:45.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:45 smithi173 ceph-mon[36632]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T12:19:45.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:45 smithi173 ceph-mon[36632]: pgmap v352: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-03-16T12:19:45.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:45 smithi098 ceph-mon[31633]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T12:19:45.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:45 smithi098 ceph-mon[31633]: pgmap v352: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-03-16T12:19:46.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:45 smithi153 ceph-mon[36901]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T12:19:46.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:45 smithi153 ceph-mon[36901]: pgmap v352: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-03-16T12:19:47.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:46 smithi098 ceph-mon[31633]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-16T12:19:47.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:46 smithi153 ceph-mon[36901]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-16T12:19:47.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:46 smithi173 ceph-mon[36632]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 -- 172.21.15.98:0/3762358714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 msgr2=0x7fdb041075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 --2- 172.21.15.98:0/3762358714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb041075f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009990 tx=0x7fdaf8009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 -- 172.21.15.98:0/3762358714 shutdown_connections 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 --2- 172.21.15.98:0/3762358714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb041075f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 --2- 172.21.15.98:0/3762358714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb04101b70 0x7fdb0410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 --2- 172.21.15.98:0/3762358714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb041011d0 0x7fdb04101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 -- 172.21.15.98:0/3762358714 >> 172.21.15.98:0/3762358714 conn(0x7fdb04078100 msgr2=0x7fdb04078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 -- 172.21.15.98:0/3762358714 shutdown_connections 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.180+0000 7fdb0c600700 1 -- 172.21.15.98:0/3762358714 wait complete. 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 Processor -- start 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 -- start start 2024-03-16T12:19:47.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 0x7fdb041a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb04101b70 0x7fdb041a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb0419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb041114a0 con 0x7fdb041011d0 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04111320 con 0x7fdb04107210 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.181+0000 7fdb0c600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04111620 con 0x7fdb04101b70 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 0x7fdb041a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 0x7fdb041a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43456/0 (socket says 172.21.15.98:43456) 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb0a600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb0419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 -- 172.21.15.98:0/4185969143 learned_addr learned my addr 172.21.15.98:0/4185969143 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 -- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb04101b70 msgr2=0x7fdb041a0e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb04101b70 0x7fdb041a0e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 -- 172.21.15.98:0/4185969143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 msgr2=0x7fdb0419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb0419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0419b240 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb0a600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb04107210 0x7fdb0419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.182+0000 7fdb09c00700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 0x7fdb041a0920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8005910 tx=0x7fdaf800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.183+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8027070 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.183+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0419b4d0 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.183+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaf8003d70 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.183+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf800f6c0 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.183+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb041a77f0 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.184+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb0406aa60 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.186+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdaf802a070 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb02000700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdaec0a29a0 0x7fdaec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdaf80d0730 con 0x7fdb041011d0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb09200700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdaec0a29a0 0x7fdaec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb02000700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] conn(0x7fdaec0adb00 0x7fdaec0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdaf8022840 con 0x7fdaec0adb00 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.187+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fdaf80d0ae0 con 0x7fdb041011d0 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.188+0000 7fdb0a600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] conn(0x7fdaec0adb00 0x7fdaec0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.188+0000 7fdb09200700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdaec0a29a0 0x7fdaec0a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb04078520 tx=0x7fdaf0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.188+0000 7fdb0a600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] conn(0x7fdaec0adb00 0x7fdaec0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.189+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== osd.2 v2:172.21.15.173:6800/2055802330 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf8022840 con 0x7fdaec0adb00 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.207+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 --> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb041028c0 con 0x7fdaec0adb00 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.207+0000 7fdb02000700 1 -- 172.21.15.98:0/4185969143 <== osd.2 v2:172.21.15.173:6800/2055802330 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb041028c0 con 0x7fdaec0adb00 2024-03-16T12:19:47.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] conn(0x7fdaec0adb00 msgr2=0x7fdaec0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] conn(0x7fdaec0adb00 0x7fdaec0affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdaec0a29a0 msgr2=0x7fdaec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdaec0a29a0 0x7fdaec0a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb04078520 tx=0x7fdaf0009260 comp rx=0 tx=0).stop 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 msgr2=0x7fdb041a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 --2- 172.21.15.98:0/4185969143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb041011d0 0x7fdb041a0920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8005910 tx=0x7fdaf800bab0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0a600700 1 -- 172.21.15.98:0/4185969143 reap_dead start 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 shutdown_connections 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 >> 172.21.15.98:0/4185969143 conn(0x7fdb04078100 msgr2=0x7fdb0406a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 shutdown_connections 2024-03-16T12:19:47.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.208+0000 7fdb0c600700 1 -- 172.21.15.98:0/4185969143 wait complete. 2024-03-16T12:19:47.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 -- 172.21.15.98:0/459525650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1406a8a0 msgr2=0x7efc1410e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 --2- 172.21.15.98:0/459525650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1406a8a0 0x7efc1410e840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc00009990 tx=0x7efc00009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 -- 172.21.15.98:0/459525650 shutdown_connections 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 --2- 172.21.15.98:0/459525650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc1410ed80 0x7efc14111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 --2- 172.21.15.98:0/459525650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14069f80 0x7efc1406a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 --2- 172.21.15.98:0/459525650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1406a8a0 0x7efc1410e840 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 -- 172.21.15.98:0/459525650 >> 172.21.15.98:0/459525650 conn(0x7efc14078100 msgr2=0x7efc14078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 -- 172.21.15.98:0/459525650 shutdown_connections 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.799+0000 7efc1b600700 1 -- 172.21.15.98:0/459525650 wait complete. 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.800+0000 7efc1b600700 1 Processor -- start 2024-03-16T12:19:47.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 -- start start 2024-03-16T12:19:47.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 0x7efc141a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1406a8a0 0x7efc141a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1410ed80 0x7efc1419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc141147c0 con 0x7efc1406a8a0 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc14114640 con 0x7efc14069f80 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc1b600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc14114940 con 0x7efc1410ed80 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc18c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 0x7efc141a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc18c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 0x7efc141a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:38534/0 (socket says 172.21.15.98:38534) 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc18c00700 1 -- 172.21.15.98:0/577842429 learned_addr learned my addr 172.21.15.98:0/577842429 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc19600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1410ed80 0x7efc1419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.801+0000 7efc13e00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1406a8a0 0x7efc141a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 -- 172.21.15.98:0/577842429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1410ed80 msgr2=0x7efc1419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1410ed80 0x7efc1419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 -- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1406a8a0 msgr2=0x7efc141a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc1406a8a0 0x7efc141a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1419b440 con 0x7efc14069f80 2024-03-16T12:19:47.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc19600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc1410ed80 0x7efc1419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc18c00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 0x7efc141a0a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efc0800e9a0 tx=0x7efc0800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0800ca10 con 0x7efc14069f80 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc1419b730 con 0x7efc14069f80 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc0800cbb0 con 0x7efc14069f80 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08018680 con 0x7efc14069f80 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.802+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc141a79c0 con 0x7efc14069f80 2024-03-16T12:19:47.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.803+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc140629e0 con 0x7efc14069f80 2024-03-16T12:19:47.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.805+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efc08010430 con 0x7efc14069f80 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc11600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efbfc0a2920 0x7efbfc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efc080cfa00 con 0x7efc14069f80 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc13e00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efbfc0a2920 0x7efbfc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc11600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] conn(0x7efbfc0ada80 0x7efbfc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc0801bec0 con 0x7efbfc0ada80 2024-03-16T12:19:47.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7efc080cfdb0 con 0x7efc14069f80 2024-03-16T12:19:47.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.807+0000 7efc19600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] conn(0x7efbfc0ada80 0x7efbfc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:47.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.808+0000 7efc13e00700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efbfc0a2920 0x7efbfc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc000098e0 tx=0x7efc00005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.808+0000 7efc19600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] conn(0x7efbfc0ada80 0x7efbfc0aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:47.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.809+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== osd.4 v2:172.21.15.153:6808/3184279467 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc0801bec0 con 0x7efbfc0ada80 2024-03-16T12:19:47.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.826+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 --> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc14062e60 con 0x7efbfc0ada80 2024-03-16T12:19:47.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.826+0000 7efc11600700 1 -- 172.21.15.98:0/577842429 <== osd.4 v2:172.21.15.153:6808/3184279467 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efc14062e60 con 0x7efbfc0ada80 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.826+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] conn(0x7efbfc0ada80 msgr2=0x7efbfc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.826+0000 7efc1b600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] conn(0x7efbfc0ada80 0x7efbfc0aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efbfc0a2920 msgr2=0x7efbfc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efbfc0a2920 0x7efbfc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc000098e0 tx=0x7efc00005960 comp rx=0 tx=0).stop 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 msgr2=0x7efc141a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 --2- 172.21.15.98:0/577842429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc14069f80 0x7efc141a0a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efc0800e9a0 tx=0x7efc0800ecb0 comp rx=0 tx=0).stop 2024-03-16T12:19:47.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc19600700 1 -- 172.21.15.98:0/577842429 reap_dead start 2024-03-16T12:19:47.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 shutdown_connections 2024-03-16T12:19:47.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 >> 172.21.15.98:0/577842429 conn(0x7efc14078100 msgr2=0x7efc14110270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:47.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 shutdown_connections 2024-03-16T12:19:47.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:47.827+0000 7efc1b600700 1 -- 172.21.15.98:0/577842429 wait complete. 2024-03-16T12:19:48.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:47 smithi098 ceph-mon[31633]: pgmap v353: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-16T12:19:48.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:47 smithi153 ceph-mon[36901]: pgmap v353: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-16T12:19:48.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:47 smithi173 ceph-mon[36632]: pgmap v353: 129 pgs: 8 peering, 121 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-16T12:19:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 -- 172.21.15.98:0/641218778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 msgr2=0x7f3c2410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 --2- 172.21.15.98:0/641218778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c2410f1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009f90 tx=0x7f3c1c00b330 comp rx=0 tx=0).stop 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 -- 172.21.15.98:0/641218778 shutdown_connections 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 --2- 172.21.15.98:0/641218778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c2410f1c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 --2- 172.21.15.98:0/641218778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102510 0x7f3c241028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 --2- 172.21.15.98:0/641218778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c24102ec0 0x7f3c24106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 -- 172.21.15.98:0/641218778 >> 172.21.15.98:0/641218778 conn(0x7f3c240fdd80 msgr2=0x7f3c241001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 -- 172.21.15.98:0/641218778 shutdown_connections 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.354+0000 7f3c2a000700 1 -- 172.21.15.98:0/641218778 wait complete. 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.355+0000 7f3c2a000700 1 Processor -- start 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 -- start start 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c24102510 0x7f3c2419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102ec0 0x7f3c2419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c241988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24112630 con 0x7f3c24102510 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c241124b0 con 0x7f3c241075f0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c2a000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c241127b0 con 0x7f3c24102ec0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c23400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c24102510 0x7f3c2419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c22a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102ec0 0x7f3c2419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c23e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c241988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c22a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102ec0 0x7f3c2419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59360/0 (socket says 172.21.15.98:59360) 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.356+0000 7f3c22a00700 1 -- 172.21.15.98:0/3944894917 learned_addr learned my addr 172.21.15.98:0/3944894917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 -- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102ec0 msgr2=0x7f3c2419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c24102ec0 0x7f3c2419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 -- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c24102510 msgr2=0x7f3c2419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c24102510 0x7f3c2419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c24199030 con 0x7f3c241075f0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c23e00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c241988d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009790 tx=0x7f3c1c00bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c021070 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c1c00be10 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c241992c0 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.357+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c005040 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.358+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c241a5590 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.358+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c24062a60 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.360+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3c1c024070 con 0x7f3c241075f0 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c1be00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c080a29a0 0x7f3c080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c1c00f020 con 0x7f3c241075f0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c23400700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c080a29a0 0x7f3c080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c1be00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] conn(0x7f3c080adb00 0x7f3c080affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c080b0670 con 0x7f3c080adb00 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3c1c0d0760 con 0x7f3c241075f0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.362+0000 7f3c22a00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] conn(0x7f3c080adb00 0x7f3c080affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.363+0000 7f3c23400700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c080a29a0 0x7f3c080a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3c240fe1d0 tx=0x7f3c14009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.363+0000 7f3c22a00700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] conn(0x7f3c080adb00 0x7f3c080affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.364+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== osd.8 v2:172.21.15.173:6816/785354592 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c080b0670 con 0x7f3c080adb00 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.382+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 --> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c24062e90 con 0x7f3c080adb00 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.382+0000 7f3c1be00700 1 -- 172.21.15.98:0/3944894917 <== osd.8 v2:172.21.15.173:6816/785354592 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3c24062e90 con 0x7f3c080adb00 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.382+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] conn(0x7f3c080adb00 msgr2=0x7f3c080affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.382+0000 7f3c2a000700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] conn(0x7f3c080adb00 0x7f3c080affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c080a29a0 msgr2=0x7f3c080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c080a29a0 0x7f3c080a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3c240fe1d0 tx=0x7f3c14009310 comp rx=0 tx=0).stop 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 msgr2=0x7f3c241988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 --2- 172.21.15.98:0/3944894917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c241075f0 0x7f3c241988d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009790 tx=0x7f3c1c00bc20 comp rx=0 tx=0).stop 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c23e00700 1 -- 172.21.15.98:0/3944894917 reap_dead start 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 shutdown_connections 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 >> 172.21.15.98:0/3944894917 conn(0x7f3c240fdd80 msgr2=0x7f3c240fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 shutdown_connections 2024-03-16T12:19:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:49.383+0000 7f3c2a000700 1 -- 172.21.15.98:0/3944894917 wait complete. 2024-03-16T12:19:50.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:49 smithi098 ceph-mon[31633]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:50.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:49 smithi153 ceph-mon[36901]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:50.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:49 smithi173 ceph-mon[36632]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:51.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:50 smithi098 ceph-mon[31633]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-16T12:19:51.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:50 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:19:51.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:50 smithi153 ceph-mon[36901]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-16T12:19:51.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:50 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:19:51.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:50 smithi173 ceph-mon[36632]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-16T12:19:51.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:50 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:19:51.546 INFO:teuthology.orchestra.run.smithi098.stdout:253403070508 2024-03-16T12:19:51.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T12:19:52.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:51 smithi098 ceph-mon[31633]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:52.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:51 smithi153 ceph-mon[36901]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:52.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:51 smithi173 ceph-mon[36632]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:53.217 INFO:teuthology.orchestra.run.smithi098.stdout:77309411407 2024-03-16T12:19:53.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T12:19:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 -- 172.21.15.98:0/1099158782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101e10 msgr2=0x7ff30c105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 --2- 172.21.15.98:0/1099158782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101e10 0x7ff30c105dd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009990 tx=0x7ff2f8009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 -- 172.21.15.98:0/1099158782 shutdown_connections 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 --2- 172.21.15.98:0/1099158782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff30c1064b0 0x7ff30c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 --2- 172.21.15.98:0/1099158782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff30c101460 0x7ff30c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 --2- 172.21.15.98:0/1099158782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101e10 0x7ff30c105dd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 -- 172.21.15.98:0/1099158782 >> 172.21.15.98:0/1099158782 conn(0x7ff30c077220 msgr2=0x7ff30c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 -- 172.21.15.98:0/1099158782 shutdown_connections 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.340+0000 7ff313a00700 1 -- 172.21.15.98:0/1099158782 wait complete. 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.341+0000 7ff313a00700 1 Processor -- start 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 -- start start 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 0x7ff30c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff30c101e10 0x7ff30c1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff30c1064b0 0x7ff30c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c111470 con 0x7ff30c1064b0 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c1112f0 con 0x7ff30c101460 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff313a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c1115f0 con 0x7ff30c101e10 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff311000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 0x7ff30c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff311000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 0x7ff30c1a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:38572/0 (socket says 172.21.15.98:38572) 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff311000700 1 -- 172.21.15.98:0/703028246 learned_addr learned my addr 172.21.15.98:0/703028246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff311a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff30c1064b0 0x7ff30c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.342+0000 7ff30be00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff30c101e10 0x7ff30c1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 -- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff30c101e10 msgr2=0x7ff30c1a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff30c101e10 0x7ff30c1a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 -- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff30c1064b0 msgr2=0x7ff30c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff30c1064b0 0x7ff30c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c19b230 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff311000700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 0x7ff30c1a0910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff30000ef10 tx=0x7ff30000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30000c9b0 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff300019070 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30c19b4c0 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff300007e60 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.343+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff30c1a77e0 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.344+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff30c0629e0 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.346+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff300012050 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff309600700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff2f40a2920 0x7ff2f40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3000cf9c0 con 0x7ff30c101460 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff30be00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff2f40a2920 0x7ff2f40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff309600700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] conn(0x7ff2f40ada80 0x7ff2f40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff300008ec0 con 0x7ff2f40ada80 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ff3000cfd70 con 0x7ff30c101460 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.348+0000 7ff311a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] conn(0x7ff2f40ada80 0x7ff2f40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.349+0000 7ff30be00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff2f40a2920 0x7ff2f40a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800b2e0 tx=0x7ff2f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.349+0000 7ff311a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] conn(0x7ff2f40ada80 0x7ff2f40aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.350+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== osd.11 v2:172.21.15.173:6824/567612982 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff300008ec0 con 0x7ff2f40ada80 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.368+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 --> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff30c062de0 con 0x7ff2f40ada80 2024-03-16T12:19:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.368+0000 7ff309600700 1 -- 172.21.15.98:0/703028246 <== osd.11 v2:172.21.15.173:6824/567612982 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff30c062de0 con 0x7ff2f40ada80 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] conn(0x7ff2f40ada80 msgr2=0x7ff2f40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] conn(0x7ff2f40ada80 0x7ff2f40aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff2f40a2920 msgr2=0x7ff2f40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff2f40a2920 0x7ff2f40a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800b2e0 tx=0x7ff2f8005960 comp rx=0 tx=0).stop 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 msgr2=0x7ff30c1a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 --2- 172.21.15.98:0/703028246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff30c101460 0x7ff30c1a0910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff30000ef10 tx=0x7ff30000c3a0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff311a00700 1 -- 172.21.15.98:0/703028246 reap_dead start 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 shutdown_connections 2024-03-16T12:19:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 >> 172.21.15.98:0/703028246 conn(0x7ff30c077220 msgr2=0x7ff30c06ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 shutdown_connections 2024-03-16T12:19:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.369+0000 7ff313a00700 1 -- 172.21.15.98:0/703028246 wait complete. 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 -- 172.21.15.98:0/529915362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074f90 msgr2=0x7efdcc112d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 --2- 172.21.15.98:0/529915362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074f90 0x7efdcc112d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009990 tx=0x7efdb8009ca0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 -- 172.21.15.98:0/529915362 shutdown_connections 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 --2- 172.21.15.98:0/529915362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc074670 0x7efdcc074a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 --2- 172.21.15.98:0/529915362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc1132c0 0x7efdcc1156b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 --2- 172.21.15.98:0/529915362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074f90 0x7efdcc112d80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.728+0000 7efdd4400700 1 -- 172.21.15.98:0/529915362 >> 172.21.15.98:0/529915362 conn(0x7efdcc0fdd80 msgr2=0x7efdcc1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.729+0000 7efdd4400700 1 -- 172.21.15.98:0/529915362 shutdown_connections 2024-03-16T12:19:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.729+0000 7efdd4400700 1 -- 172.21.15.98:0/529915362 wait complete. 2024-03-16T12:19:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.730+0000 7efdd4400700 1 Processor -- start 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 -- start start 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 0x7efdcc19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 0x7efdcc19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 0x7efdcc1a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc118c00 con 0x7efdcc074670 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc118a80 con 0x7efdcc1132c0 2024-03-16T12:19:53.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd4400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc118d80 con 0x7efdcc074f90 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd2400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 0x7efdcc1a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd1a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 0x7efdcc19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd1000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 0x7efdcc19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd2400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 0x7efdcc1a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:38610/0 (socket says 172.21.15.98:38610) 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd1a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 0x7efdcc19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43536/0 (socket says 172.21.15.98:43536) 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd1000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 0x7efdcc19f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59412/0 (socket says 172.21.15.98:59412) 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.731+0000 7efdd2400700 1 -- 172.21.15.98:0/37019364 learned_addr learned my addr 172.21.15.98:0/37019364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:53.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 -- 172.21.15.98:0/37019364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 msgr2=0x7efdcc19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 0x7efdcc19f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 -- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 msgr2=0x7efdcc19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 0x7efdcc19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc1a4080 con 0x7efdcc1132c0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd1a00700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc074670 0x7efdcc19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd1000700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdcc074f90 0x7efdcc19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdd2400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 0x7efdcc1a3940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efdc400ef90 tx=0x7efdc400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.732+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc40092e0 con 0x7efdcc1132c0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.733+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc4004bb0 con 0x7efdcc1132c0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.733+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc1a4310 con 0x7efdcc1132c0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.733+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4004280 con 0x7efdcc1132c0 2024-03-16T12:19:53.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.733+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1abbf0 con 0x7efdcc1132c0 2024-03-16T12:19:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.734+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efdcc062a60 con 0x7efdcc1132c0 2024-03-16T12:19:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.736+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efdc401b030 con 0x7efdcc1132c0 2024-03-16T12:19:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdca000700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efdb40a2920 0x7efdb40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efdc40cf3d0 con 0x7efdcc1132c0 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdd1a00700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efdb40a2920 0x7efdb40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdca000700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] conn(0x7efdb40ada80 0x7efdb40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdb40b05f0 con 0x7efdb40ada80 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.738+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7efdc40cf780 con 0x7efdcc1132c0 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.739+0000 7efdd1000700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] conn(0x7efdb40ada80 0x7efdb40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.739+0000 7efdd1a00700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efdb40a2920 0x7efdb40a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdc0011490 tx=0x7efdc0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.739+0000 7efdd1000700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] conn(0x7efdb40ada80 0x7efdb40aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.740+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== osd.9 v2:172.21.15.98:6826/3340262807 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efdb40b05f0 con 0x7efdb40ada80 2024-03-16T12:19:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.758+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 --> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efdcc062ee0 con 0x7efdb40ada80 2024-03-16T12:19:53.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.758+0000 7efdca000700 1 -- 172.21.15.98:0/37019364 <== osd.9 v2:172.21.15.98:6826/3340262807 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efdcc062ee0 con 0x7efdb40ada80 2024-03-16T12:19:53.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] conn(0x7efdb40ada80 msgr2=0x7efdb40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] conn(0x7efdb40ada80 0x7efdb40aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efdb40a2920 msgr2=0x7efdb40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efdb40a2920 0x7efdb40a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdc0011490 tx=0x7efdc0009260 comp rx=0 tx=0).stop 2024-03-16T12:19:53.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 msgr2=0x7efdcc1a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 --2- 172.21.15.98:0/37019364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdcc1132c0 0x7efdcc1a3940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efdc400ef90 tx=0x7efdc400c3a0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd2400700 1 -- 172.21.15.98:0/37019364 reap_dead start 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 shutdown_connections 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 >> 172.21.15.98:0/37019364 conn(0x7efdcc0fdd80 msgr2=0x7efdcc0fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 shutdown_connections 2024-03-16T12:19:53.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:53.759+0000 7efdd4400700 1 -- 172.21.15.98:0/37019364 wait complete. 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 -- 172.21.15.98:0/2716556954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8108320 msgr2=0x7fa0c8108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 --2- 172.21.15.98:0/2716556954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8108320 0x7fa0c8108700 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009920 tx=0x7fa0bc009c30 comp rx=0 tx=0).stop 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 -- 172.21.15.98:0/2716556954 shutdown_connections 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 --2- 172.21.15.98:0/2716556954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0c806a920 0x7fa0c810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 --2- 172.21.15.98:0/2716556954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8108320 0x7fa0c8108700 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 --2- 172.21.15.98:0/2716556954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c8069f80 0x7fa0c806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 -- 172.21.15.98:0/2716556954 >> 172.21.15.98:0/2716556954 conn(0x7fa0c8078100 msgr2=0x7fa0c8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 -- 172.21.15.98:0/2716556954 shutdown_connections 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.854+0000 7fa0d0800700 1 -- 172.21.15.98:0/2716556954 wait complete. 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.855+0000 7fa0d0800700 1 Processor -- start 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.855+0000 7fa0d0800700 1 -- start start 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.855+0000 7fa0d0800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8069f80 0x7fa0c81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.855+0000 7fa0d0800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 0x7fa0c81a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0d0800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0c8108320 0x7fa0c819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0d0800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c8112580 con 0x7fa0c8069f80 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0d0800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c8112400 con 0x7fa0c806a920 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0d0800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c8112700 con 0x7fa0c8108320 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 0x7fa0c81a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cde00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8069f80 0x7fa0c81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 0x7fa0c81a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:38588/0 (socket says 172.21.15.98:38588) 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 -- 172.21.15.98:0/331245546 learned_addr learned my addr 172.21.15.98:0/331245546 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0ce800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0c8108320 0x7fa0c819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 -- 172.21.15.98:0/331245546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0c8108320 msgr2=0x7fa0c819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0c8108320 0x7fa0c819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 -- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8069f80 msgr2=0x7fa0c81a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0c8069f80 0x7fa0c81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c819b260 con 0x7fa0c806a920 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.856+0000 7fa0cd400700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 0x7fa0c81a0e80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9a0 tx=0x7fa0b800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.857+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800ca10 con 0x7fa0c806a920 2024-03-16T12:19:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.857+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0b800cbb0 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.857+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c819b550 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.857+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8018680 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.857+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c81a7810 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.858+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0c810df10 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.860+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa0b8047a90 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.861+0000 7fa0c2a00700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa0b00a29a0 0x7fa0b00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.861+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa0b80cf340 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0cde00700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa0b00a29a0 0x7fa0b00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0c2a00700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] conn(0x7fa0b00ada80 0x7fa0b00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0b8015e80 con 0x7fa0b00ada80 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa0b80cf6f0 con 0x7fa0c806a920 2024-03-16T12:19:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0ce800700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] conn(0x7fa0b00ada80 0x7fa0b00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0cde00700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa0b00a29a0 0x7fa0b00a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c010 tx=0x7fa0bc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.862+0000 7fa0ce800700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] conn(0x7fa0b00ada80 0x7fa0b00aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.864+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== osd.7 v2:172.21.15.153:6816/838772754 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0b8015e80 con 0x7fa0b00ada80 2024-03-16T12:19:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.882+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 --> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa0c81039d0 con 0x7fa0b00ada80 2024-03-16T12:19:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.882+0000 7fa0c2a00700 1 -- 172.21.15.98:0/331245546 <== osd.7 v2:172.21.15.153:6816/838772754 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa0c81039d0 con 0x7fa0b00ada80 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] conn(0x7fa0b00ada80 msgr2=0x7fa0b00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] conn(0x7fa0b00ada80 0x7fa0b00aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa0b00a29a0 msgr2=0x7fa0b00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa0b00a29a0 0x7fa0b00a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c010 tx=0x7fa0bc019040 comp rx=0 tx=0).stop 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 msgr2=0x7fa0c81a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 --2- 172.21.15.98:0/331245546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c806a920 0x7fa0c81a0e80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9a0 tx=0x7fa0b800ecb0 comp rx=0 tx=0).stop 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0ce800700 1 -- 172.21.15.98:0/331245546 reap_dead start 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 shutdown_connections 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 >> 172.21.15.98:0/331245546 conn(0x7fa0c8078100 msgr2=0x7fa0c810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 shutdown_connections 2024-03-16T12:19:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:52.883+0000 7fa0d0800700 1 -- 172.21.15.98:0/331245546 wait complete. 2024-03-16T12:19:53.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:53 smithi098 ceph-mon[31633]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:54.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:53 smithi153 ceph-mon[36901]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:54.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:53 smithi173 ceph-mon[36632]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-03-16T12:19:55.088 INFO:teuthology.orchestra.run.smithi098.stdout:141733920836 2024-03-16T12:19:55.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T12:19:55.234 INFO:teuthology.orchestra.run.smithi098.stdout:219043332147 2024-03-16T12:19:55.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T12:19:55.435 INFO:teuthology.orchestra.run.smithi098.stdout:163208757308 2024-03-16T12:19:55.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T12:19:56.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:55 smithi098 ceph-mon[31633]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:56.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:55 smithi153 ceph-mon[36901]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:56.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:55 smithi173 ceph-mon[36632]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:56.580 INFO:teuthology.orchestra.run.smithi098.stdout:287762808871 2024-03-16T12:19:56.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T12:19:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 -- 172.21.15.98:0/3983569580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f84102350 msgr2=0x7f3f841027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 --2- 172.21.15.98:0/3983569580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f84102350 0x7f3f841027d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f700098f0 tx=0x7f3f70009c00 comp rx=0 tx=0).stop 2024-03-16T12:19:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 -- 172.21.15.98:0/3983569580 shutdown_connections 2024-03-16T12:19:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 --2- 172.21.15.98:0/3983569580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f841083a0 0x7f3f84108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 --2- 172.21.15.98:0/3983569580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 0x7f3f8410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 --2- 172.21.15.98:0/3983569580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f84102350 0x7f3f841027d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 -- 172.21.15.98:0/3983569580 >> 172.21.15.98:0/3983569580 conn(0x7f3f84079420 msgr2=0x7f3f84079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 -- 172.21.15.98:0/3983569580 shutdown_connections 2024-03-16T12:19:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.130+0000 7f3f89400700 1 -- 172.21.15.98:0/3983569580 wait complete. 2024-03-16T12:19:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.132+0000 7f3f89400700 1 Processor -- start 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.132+0000 7f3f89400700 1 -- start start 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 0x7f3f8419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 0x7f3f8419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f841083a0 0x7f3f84198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f841125c0 con 0x7f3f841083a0 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f84112440 con 0x7f3f84102350 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f89400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f84112740 con 0x7f3f84102d10 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f82800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 0x7f3f8419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 0x7f3f8419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f83200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f841083a0 0x7f3f84198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f82800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 0x7f3f8419e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:48402/0 (socket says 172.21.15.98:48402) 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f82800700 1 -- 172.21.15.98:0/4160514664 learned_addr learned my addr 172.21.15.98:0/4160514664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 -- 172.21.15.98:0/4160514664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 msgr2=0x7f3f8419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 0x7f3f8419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 -- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f841083a0 msgr2=0x7f3f84198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f841083a0 0x7f3f84198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.133+0000 7f3f81e00700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f84199060 con 0x7f3f84102d10 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f82800700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f84102350 0x7f3f8419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f83200700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f841083a0 0x7f3f84198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:19:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f81e00700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 0x7f3f8419ec30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000c00 tx=0x7f3f7000ff20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70005490 con 0x7f3f84102d10 2024-03-16T12:19:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f70005630 con 0x7f3f84102d10 2024-03-16T12:19:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f841992f0 con 0x7f3f84102d10 2024-03-16T12:19:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70003d90 con 0x7f3f84102d10 2024-03-16T12:19:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.134+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f841a55c0 con 0x7f3f84102d10 2024-03-16T12:19:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.135+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f84062a60 con 0x7f3f84102d10 2024-03-16T12:19:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.137+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3f7000b560 con 0x7f3f84102d10 2024-03-16T12:19:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f7b400700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3f680a29a0 0x7f3f680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3f700cfca0 con 0x7f3f84102d10 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f82800700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3f680a29a0 0x7f3f680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f7b400700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] conn(0x7f3f680ad9d0 0x7f3f680afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f680b0540 con 0x7f3f680ad9d0 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.139+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3f700d0050 con 0x7f3f84102d10 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.140+0000 7f3f83200700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] conn(0x7f3f680ad9d0 0x7f3f680afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.140+0000 7f3f82800700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3f680a29a0 0x7f3f680a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009750 tx=0x7f3f74009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:57.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.140+0000 7f3f83200700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] conn(0x7f3f680ad9d0 0x7f3f680afe70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:57.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.142+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== osd.3 v2:172.21.15.98:6810/2502573186 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f680b0540 con 0x7f3f680ad9d0 2024-03-16T12:19:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.160+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 --> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3f84062e60 con 0x7f3f680ad9d0 2024-03-16T12:19:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.160+0000 7f3f7b400700 1 -- 172.21.15.98:0/4160514664 <== osd.3 v2:172.21.15.98:6810/2502573186 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3f84062e60 con 0x7f3f680ad9d0 2024-03-16T12:19:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] conn(0x7f3f680ad9d0 msgr2=0x7f3f680afe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] conn(0x7f3f680ad9d0 0x7f3f680afe70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3f680a29a0 msgr2=0x7f3f680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3f680a29a0 0x7f3f680a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009750 tx=0x7f3f74009310 comp rx=0 tx=0).stop 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 msgr2=0x7f3f8419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 --2- 172.21.15.98:0/4160514664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f84102d10 0x7f3f8419ec30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000c00 tx=0x7f3f7000ff20 comp rx=0 tx=0).stop 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f83200700 1 -- 172.21.15.98:0/4160514664 reap_dead start 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 shutdown_connections 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 >> 172.21.15.98:0/4160514664 conn(0x7f3f84079420 msgr2=0x7f3f841012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 shutdown_connections 2024-03-16T12:19:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:57.161+0000 7f3f89400700 1 -- 172.21.15.98:0/4160514664 wait complete. 2024-03-16T12:19:57.916 INFO:teuthology.orchestra.run.smithi098.stdout:343597383709 2024-03-16T12:19:57.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T12:19:58.133 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:57 smithi098 ceph-mon[31633]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:58.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:57 smithi153 ceph-mon[36901]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:58.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:57 smithi173 ceph-mon[36632]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 -- 172.21.15.98:0/2686009888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec341088e0 msgr2=0x7fec34108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 --2- 172.21.15.98:0/2686009888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec341088e0 0x7fec34108d60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fec200098f0 tx=0x7fec20009c00 comp rx=0 tx=0).stop 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 -- 172.21.15.98:0/2686009888 shutdown_connections 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 --2- 172.21.15.98:0/2686009888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 0x7fec341157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 --2- 172.21.15.98:0/2686009888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec3410e8e0 0x7fec3410ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 --2- 172.21.15.98:0/2686009888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec341088e0 0x7fec34108d60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 -- 172.21.15.98:0/2686009888 >> 172.21.15.98:0/2686009888 conn(0x7fec34075aa0 msgr2=0x7fec34077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 -- 172.21.15.98:0/2686009888 shutdown_connections 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.325+0000 7fec3b600700 1 -- 172.21.15.98:0/2686009888 wait complete. 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.326+0000 7fec3b600700 1 Processor -- start 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 -- start start 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec341088e0 0x7fec3419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 0x7fec3419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec3410e8e0 0x7fec341a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec34118c60 con 0x7fec341088e0 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec34118ae0 con 0x7fec341092a0 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec3b600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec34118de0 con 0x7fec3410e8e0 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec38c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec341088e0 0x7fec3419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec33e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 0x7fec3419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec38c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec341088e0 0x7fec3419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57240/0 (socket says 172.21.15.98:57240) 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec38c00700 1 -- 172.21.15.98:0/660680177 learned_addr learned my addr 172.21.15.98:0/660680177 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.327+0000 7fec39600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec3410e8e0 0x7fec341a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 -- 172.21.15.98:0/660680177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec3410e8e0 msgr2=0x7fec341a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec3410e8e0 0x7fec341a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 -- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec341088e0 msgr2=0x7fec3419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec341088e0 0x7fec3419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec341a40c0 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec33e00700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 0x7fec3419f5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fec20000c00 tx=0x7fec2000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec20005040 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec200056c0 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec20003a00 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec341a4350 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.328+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec341abc30 con 0x7fec341092a0 2024-03-16T12:19:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.329+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec341a47e0 con 0x7fec341092a0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.331+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fec2000b560 con 0x7fec341092a0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.332+0000 7fec31600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fec1c0a29a0 0x7fec1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec200cffe0 con 0x7fec341092a0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec38c00700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fec1c0a29a0 0x7fec1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec31600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] conn(0x7fec1c0adb00 0x7fec1c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec2000bde0 con 0x7fec1c0adb00 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fec200d03f0 con 0x7fec341092a0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec39600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] conn(0x7fec1c0adb00 0x7fec1c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.333+0000 7fec38c00700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fec1c0a29a0 0x7fec1c0a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec28009ca0 tx=0x7fec28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.334+0000 7fec39600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] conn(0x7fec1c0adb00 0x7fec1c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.335+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== osd.1 v2:172.21.15.153:6800/4082140415 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec2000bde0 con 0x7fec1c0adb00 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.353+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 --> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec34062d80 con 0x7fec1c0adb00 2024-03-16T12:19:58.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.353+0000 7fec31600700 1 -- 172.21.15.98:0/660680177 <== osd.1 v2:172.21.15.153:6800/4082140415 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fec34062d80 con 0x7fec1c0adb00 2024-03-16T12:19:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] conn(0x7fec1c0adb00 msgr2=0x7fec1c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] conn(0x7fec1c0adb00 0x7fec1c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:19:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fec1c0a29a0 msgr2=0x7fec1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fec1c0a29a0 0x7fec1c0a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec28009ca0 tx=0x7fec28009260 comp rx=0 tx=0).stop 2024-03-16T12:19:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 msgr2=0x7fec3419f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 --2- 172.21.15.98:0/660680177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec341092a0 0x7fec3419f5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fec20000c00 tx=0x7fec2000fa00 comp rx=0 tx=0).stop 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec39600700 1 -- 172.21.15.98:0/660680177 reap_dead start 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 shutdown_connections 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 >> 172.21.15.98:0/660680177 conn(0x7fec34075aa0 msgr2=0x7fec34076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 shutdown_connections 2024-03-16T12:19:58.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:19:58.354+0000 7fec3b600700 1 -- 172.21.15.98:0/660680177 wait complete. 2024-03-16T12:19:59.040 INFO:teuthology.orchestra.run.smithi098.stdout:107374182474 2024-03-16T12:19:59.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T12:19:59.400 INFO:teuthology.orchestra.run.smithi098.stdout:55834574934 2024-03-16T12:19:59.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T12:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:19:59.984 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.055+0000 7fbc29200700 1 -- 172.21.15.98:0/533043184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074500 msgr2=0x7fbc240748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.055+0000 7fbc29200700 1 --2- 172.21.15.98:0/533043184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074500 0x7fbc240748e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009990 tx=0x7fbc14009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 -- 172.21.15.98:0/533043184 shutdown_connections 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 --2- 172.21.15.98:0/533043184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074500 0x7fbc240748e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 --2- 172.21.15.98:0/533043184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc24115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 --2- 172.21.15.98:0/533043184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc24074e20 0x7fbc24112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 -- 172.21.15.98:0/533043184 >> 172.21.15.98:0/533043184 conn(0x7fbc24079430 msgr2=0x7fbc24079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 -- 172.21.15.98:0/533043184 shutdown_connections 2024-03-16T12:20:00.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.056+0000 7fbc29200700 1 -- 172.21.15.98:0/533043184 wait complete. 2024-03-16T12:20:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.057+0000 7fbc29200700 1 Processor -- start 2024-03-16T12:20:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.057+0000 7fbc29200700 1 -- start start 2024-03-16T12:20:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc24074500 0x7fbc2419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074e20 0x7fbc2419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc241a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24118ce0 con 0x7fbc24074500 2024-03-16T12:20:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24118b60 con 0x7fbc24074e20 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc29200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24118e60 con 0x7fbc241134a0 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc22800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc241a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc21e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc24074500 0x7fbc2419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc22800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc241a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:45238/0 (socket says 172.21.15.98:45238) 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.058+0000 7fbc22800700 1 -- 172.21.15.98:0/782678719 learned_addr learned my addr 172.21.15.98:0/782678719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc21400700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074e20 0x7fbc2419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 -- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074e20 msgr2=0x7fbc2419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074e20 0x7fbc2419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 -- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc24074500 msgr2=0x7fbc2419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc24074500 0x7fbc2419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc241a40a0 con 0x7fbc241134a0 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc21400700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc24074e20 0x7fbc2419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:20:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc22800700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc241a3960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00c0f0 tx=0x7fbc1c00c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c018020 con 0x7fbc241134a0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc1c003a00 con 0x7fbc241134a0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.059+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c004840 con 0x7fbc241134a0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.060+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc241a4390 con 0x7fbc241134a0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.060+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc241abc10 con 0x7fbc241134a0 2024-03-16T12:20:00.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.061+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbc24112b90 con 0x7fbc241134a0 2024-03-16T12:20:00.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.063+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbc1c012050 con 0x7fbc241134a0 2024-03-16T12:20:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.064+0000 7fbc1aa00700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbc080a29a0 0x7fbc080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.064+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc1c0ceb80 con 0x7fbc241134a0 2024-03-16T12:20:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.064+0000 7fbc1aa00700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] conn(0x7fbc080adb00 0x7fbc080affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.064+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc080b0670 con 0x7fbc080adb00 2024-03-16T12:20:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.064+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbc1c0d32a0 con 0x7fbc241134a0 2024-03-16T12:20:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.065+0000 7fbc21400700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] conn(0x7fbc080adb00 0x7fbc080affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.065+0000 7fbc21e00700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbc080a29a0 0x7fbc080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.066+0000 7fbc21400700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] conn(0x7fbc080adb00 0x7fbc080affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.066+0000 7fbc21e00700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbc080a29a0 0x7fbc080a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400b2e0 tx=0x7fbc14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.067+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== osd.10 v2:172.21.15.153:6824/2757700029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc080b0670 con 0x7fbc080adb00 2024-03-16T12:20:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.085+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 --> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbc24075c30 con 0x7fbc080adb00 2024-03-16T12:20:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc1aa00700 1 -- 172.21.15.98:0/782678719 <== osd.10 v2:172.21.15.153:6824/2757700029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbc24075c30 con 0x7fbc080adb00 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] conn(0x7fbc080adb00 msgr2=0x7fbc080affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] conn(0x7fbc080adb00 0x7fbc080affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbc080a29a0 msgr2=0x7fbc080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbc080a29a0 0x7fbc080a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400b2e0 tx=0x7fbc14005fd0 comp rx=0 tx=0).stop 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 msgr2=0x7fbc241a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 --2- 172.21.15.98:0/782678719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc241134a0 0x7fbc241a3960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00c0f0 tx=0x7fbc1c00c400 comp rx=0 tx=0).stop 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc22800700 1 -- 172.21.15.98:0/782678719 reap_dead start 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 shutdown_connections 2024-03-16T12:20:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.086+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 >> 172.21.15.98:0/782678719 conn(0x7fbc24079430 msgr2=0x7fbc241121f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:00.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.087+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 shutdown_connections 2024-03-16T12:20:00.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:00.087+0000 7fbc29200700 1 -- 172.21.15.98:0/782678719 wait complete. 2024-03-16T12:20:00.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:19:59 smithi098 ceph-mon[31633]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:20:00.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:19:59 smithi153 ceph-mon[36901]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:20:00.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:19:59 smithi173 ceph-mon[36632]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T12:20:01.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:00 smithi098 ceph-mon[31633]: overall HEALTH_OK 2024-03-16T12:20:01.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:00 smithi153 ceph-mon[36901]: overall HEALTH_OK 2024-03-16T12:20:01.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:00 smithi173 ceph-mon[36632]: overall HEALTH_OK 2024-03-16T12:20:02.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:01 smithi098 ceph-mon[31633]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:20:02.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:01 smithi153 ceph-mon[36901]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:20:02.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:01 smithi173 ceph-mon[36632]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:20:02.560 INFO:teuthology.orchestra.run.smithi098.stdout:309237645346 2024-03-16T12:20:02.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T12:20:04.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:03 smithi098 ceph-mon[31633]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:04.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:03 smithi153 ceph-mon[36901]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:04.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:03 smithi173 ceph-mon[36632]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:06.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:05 smithi098 ceph-mon[31633]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:06.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:05 smithi153 ceph-mon[36901]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:06.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:05 smithi173 ceph-mon[36632]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.650+0000 7ff98bc00700 1 -- 172.21.15.98:0/4218465100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 msgr2=0x7ff984112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.650+0000 7ff98bc00700 1 --2- 172.21.15.98:0/4218465100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff984112f60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff970009990 tx=0x7ff970009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 -- 172.21.15.98:0/4218465100 shutdown_connections 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 --2- 172.21.15.98:0/4218465100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9841134a0 0x7ff984115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 --2- 172.21.15.98:0/4218465100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff984074500 0x7ff9840748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 --2- 172.21.15.98:0/4218465100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff984112f60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 -- 172.21.15.98:0/4218465100 >> 172.21.15.98:0/4218465100 conn(0x7ff984079430 msgr2=0x7ff984079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 -- 172.21.15.98:0/4218465100 shutdown_connections 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.651+0000 7ff98bc00700 1 -- 172.21.15.98:0/4218465100 wait complete. 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 Processor -- start 2024-03-16T12:20:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 -- start start 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 0x7ff98419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9841134a0 0x7ff9841a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff984118cf0 con 0x7ff984074500 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff984118b70 con 0x7ff984074e20 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.652+0000 7ff98bc00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff984118e70 con 0x7ff9841134a0 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff989200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 0x7ff98419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:48436/0 (socket says 172.21.15.98:48436) 2024-03-16T12:20:06.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 -- 172.21.15.98:0/19672001 learned_addr learned my addr 172.21.15.98:0/19672001 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff989c00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9841134a0 0x7ff9841a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 -- 172.21.15.98:0/19672001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9841134a0 msgr2=0x7ff9841a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9841134a0 0x7ff9841a3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 -- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 msgr2=0x7ff98419f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 0x7ff98419f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 -- 172.21.15.98:0/19672001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9841a4170 con 0x7ff984074e20 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff989200700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 0x7ff98419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.653+0000 7ff983e00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff970000c00 tx=0x7ff970005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.654+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970027070 con 0x7ff984074e20 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.654+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff97000fda0 con 0x7ff984074e20 2024-03-16T12:20:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.654+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700173f0 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.654+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9841a4400 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.654+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9841abce0 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.655+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff984075c30 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.657+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff97002a070 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.658+0000 7ff981600700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 0x7ff96c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.659+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff9700cff10 con 0x7ff984074e20 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.659+0000 7ff989200700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 0x7ff96c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.660+0000 7ff989200700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 0x7ff96c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff978011490 tx=0x7ff978009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.663+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff970031080 con 0x7ff984074e20 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 -- 172.21.15.98:0/790213491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c076800 msgr2=0x7ff52c076be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 --2- 172.21.15.98:0/790213491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c076800 0x7ff52c076be0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff520009990 tx=0x7ff520009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 -- 172.21.15.98:0/790213491 shutdown_connections 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 --2- 172.21.15.98:0/790213491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c076800 0x7ff52c076be0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 --2- 172.21.15.98:0/790213491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 0x7ff52c115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 --2- 172.21.15.98:0/790213491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c077120 0x7ff52c112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 -- 172.21.15.98:0/790213491 >> 172.21.15.98:0/790213491 conn(0x7ff52c0fffd0 msgr2=0x7ff52c1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 -- 172.21.15.98:0/790213491 shutdown_connections 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.759+0000 7ff534800700 1 -- 172.21.15.98:0/790213491 wait complete. 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.760+0000 7ff534800700 1 Processor -- start 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 -- start start 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c076800 0x7ff52c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 0x7ff52c1a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c118ca0 con 0x7ff52c076800 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c118b20 con 0x7ff52c077120 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff534800700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c118e20 con 0x7ff52c113180 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff531400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff531400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:48452/0 (socket says 172.21.15.98:48452) 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff531400700 1 -- 172.21.15.98:0/1796424583 learned_addr learned my addr 172.21.15.98:0/1796424583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff531e00700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c076800 0x7ff52c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.761+0000 7ff532800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 0x7ff52c1a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 -- 172.21.15.98:0/1796424583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 msgr2=0x7ff52c1a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 0x7ff52c1a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 -- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c076800 msgr2=0x7ff52c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c076800 0x7ff52c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 -- 172.21.15.98:0/1796424583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c1a4100 con 0x7ff52c077120 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff531400700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9a0 tx=0x7ff51c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c00c9c0 con 0x7ff52c077120 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51c00cb60 con 0x7ff52c077120 2024-03-16T12:20:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c1a43f0 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.762+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c018660 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.763+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1abc70 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.764+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c068f90 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.766+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff51c00cd10 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.767+0000 7ff526a00700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 0x7ff5140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.767+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff51c0cf080 con 0x7ff52c077120 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.767+0000 7ff531e00700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 0x7ff5140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.768+0000 7ff531e00700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 0x7ff5140a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff520006010 tx=0x7ff52000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.769+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff51c09bc00 con 0x7ff52c077120 2024-03-16T12:20:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.902+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff9841a04c0 con 0x7ff984074e20 2024-03-16T12:20:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.902+0000 7ff981600700 1 -- 172.21.15.98:0/19672001 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff970013070 con 0x7ff984074e20 2024-03-16T12:20:06.885 INFO:teuthology.orchestra.run.smithi098.stdout:253403070511 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 msgr2=0x7ff96c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 0x7ff96c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff978011490 tx=0x7ff978009260 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 msgr2=0x7ff98419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff970000c00 tx=0x7ff970005590 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 shutdown_connections 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9841134a0 0x7ff9841a3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff984074500 0x7ff98419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff96c0a29a0 0x7ff96c0a4e60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 --2- 172.21.15.98:0/19672001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff984074e20 0x7ff98419f6a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.905+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 >> 172.21.15.98:0/19672001 conn(0x7ff984079430 msgr2=0x7ff984112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.906+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 shutdown_connections 2024-03-16T12:20:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.906+0000 7ff98bc00700 1 -- 172.21.15.98:0/19672001 wait complete. 2024-03-16T12:20:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:06.997+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff52c02db70 con 0x7ff52c077120 2024-03-16T12:20:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.000+0000 7ff526a00700 1 -- 172.21.15.98:0/1796424583 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff51c09b500 con 0x7ff52c077120 2024-03-16T12:20:06.982 INFO:teuthology.orchestra.run.smithi098.stdout:34359738461 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.003+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 msgr2=0x7ff5140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.003+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 0x7ff5140a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff520006010 tx=0x7ff52000b260 comp rx=0 tx=0).stop 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 msgr2=0x7ff52c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9a0 tx=0x7ff51c00ecb0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 shutdown_connections 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff52c113180 0x7ff52c1a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff52c076800 0x7ff52c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff5140a29a0 0x7ff5140a4e60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 --2- 172.21.15.98:0/1796424583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff52c077120 0x7ff52c19f630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 >> 172.21.15.98:0/1796424583 conn(0x7ff52c0fffd0 msgr2=0x7ff52c1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 shutdown_connections 2024-03-16T12:20:06.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:07.004+0000 7ff534800700 1 -- 172.21.15.98:0/1796424583 wait complete. 2024-03-16T12:20:07.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:07 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/19672001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T12:20:07.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:07 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/19672001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T12:20:07.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:07 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/19672001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T12:20:08.201 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:08 smithi098 ceph-mon[31633]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:08.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:08 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1796424583' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T12:20:08.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:08 smithi153 ceph-mon[36901]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:08.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:08 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1796424583' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T12:20:08.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:08 smithi173 ceph-mon[36632]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:08.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:08 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1796424583' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T12:20:09.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:09 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1988744194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T12:20:09.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:09 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1988744194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T12:20:09.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:09 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1988744194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T12:20:09.659 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070508 got 253403070511 for osd.8 2024-03-16T12:20:09.659 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:09.949 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738461 for osd.0 2024-03-16T12:20:09.949 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 -- 172.21.15.98:0/760235600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae806a940 msgr2=0x7f4ae810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 --2- 172.21.15.98:0/760235600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae806a940 0x7f4ae810f120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4ae000b0d0 tx=0x7f4ae000b3e0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 -- 172.21.15.98:0/760235600 shutdown_connections 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 --2- 172.21.15.98:0/760235600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae806a940 0x7f4ae810f120 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 --2- 172.21.15.98:0/760235600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae8108300 0x7f4ae81086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 --2- 172.21.15.98:0/760235600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 0x7f4ae806a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 -- 172.21.15.98:0/760235600 >> 172.21.15.98:0/760235600 conn(0x7f4ae8077f80 msgr2=0x7f4ae8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 -- 172.21.15.98:0/760235600 shutdown_connections 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.207+0000 7f4aee200700 1 -- 172.21.15.98:0/760235600 wait complete. 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.208+0000 7f4aee200700 1 Processor -- start 2024-03-16T12:20:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 -- start start 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 0x7f4ae81015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae806a940 0x7f4ae8101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8112580 con 0x7f4ae8069f80 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8112400 con 0x7f4ae8108300 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4aee200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae8112700 con 0x7f4ae806a940 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4ae7e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4ae7400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 0x7f4ae81015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4ae7e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:56446/0 (socket says 172.21.15.98:56446) 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4ae7e00700 1 -- 172.21.15.98:0/1988744194 learned_addr learned my addr 172.21.15.98:0/1988744194 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.209+0000 7f4adfe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae806a940 0x7f4ae8101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 -- 172.21.15.98:0/1988744194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae806a940 msgr2=0x7f4ae8101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae806a940 0x7f4ae8101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 -- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 msgr2=0x7f4ae81015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 0x7f4ae81015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 -- 172.21.15.98:0/1988744194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae81a6f90 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae7e00700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006070 tx=0x7f4ae0007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0003c00 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ae0003da0 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae81a7220 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.210+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae001b880 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.211+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae81a7620 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.212+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae8068f90 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.213+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4ae0026400 con 0x7f4ae8108300 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.214+0000 7f4ae4c00700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 0x7f4acc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:10.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.215+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4ae00d03a0 con 0x7f4ae8108300 2024-03-16T12:20:10.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.215+0000 7f4ae7400700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 0x7f4acc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:10.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.215+0000 7f4ae7400700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 0x7f4acc0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009be0 tx=0x7f4ad8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:10.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.219+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ae009d010 con 0x7f4ae8108300 2024-03-16T12:20:10.116 INFO:teuthology.orchestra.run.smithi098.stdout:77309411411 2024-03-16T12:20:10.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.460+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4ae802dd30 con 0x7f4ae8108300 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.460+0000 7f4ae4c00700 1 -- 172.21.15.98:0/1988744194 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4ae009c910 con 0x7f4ae8108300 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 msgr2=0x7f4acc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 0x7f4acc0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009be0 tx=0x7f4ad8009260 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 msgr2=0x7f4ae8102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0006070 tx=0x7f4ae0007a20 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 shutdown_connections 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4acc0a2920 0x7f4acc0a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae8108300 0x7f4ae8102070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae8069f80 0x7f4ae81015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 --2- 172.21.15.98:0/1988744194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ae806a940 0x7f4ae8101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:10.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.463+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 >> 172.21.15.98:0/1988744194 conn(0x7f4ae8077f80 msgr2=0x7f4ae8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:10.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.464+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 shutdown_connections 2024-03-16T12:20:10.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:08.464+0000 7f4aee200700 1 -- 172.21.15.98:0/1988744194 wait complete. 2024-03-16T12:20:10.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:10 smithi098 ceph-mon[31633]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:10.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:10 smithi153 ceph-mon[36901]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:10.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:10 smithi173 ceph-mon[36632]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:11.672 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:11.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411411 for osd.2 2024-03-16T12:20:11.693 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:12 smithi098 ceph-mon[31633]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:12.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:12 smithi153 ceph-mon[36901]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:12.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:12 smithi173 ceph-mon[36632]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 -- 172.21.15.98:0/2984015640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901044e0 msgr2=0x7fbb90104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2984015640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901044e0 0x7fbb90104960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009920 tx=0x7fbb78009c30 comp rx=0 tx=0).stop 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 -- 172.21.15.98:0/2984015640 shutdown_connections 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2984015640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 0x7fbb90111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2984015640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9010a590 0x7fbb9010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2984015640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901044e0 0x7fbb90104960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 -- 172.21.15.98:0/2984015640 >> 172.21.15.98:0/2984015640 conn(0x7fbb900fffb0 msgr2=0x7fbb901023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 -- 172.21.15.98:0/2984015640 shutdown_connections 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.499+0000 7fbb95a00700 1 -- 172.21.15.98:0/2984015640 wait complete. 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.500+0000 7fbb95a00700 1 Processor -- start 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.500+0000 7fbb95a00700 1 -- start start 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 0x7fbb901a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9010a590 0x7fbb9019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90114820 con 0x7fbb9010a590 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901146a0 con 0x7fbb90104ea0 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb95a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901149a0 con 0x7fbb901044e0 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38404/0 (socket says 172.21.15.98:38404) 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 -- 172.21.15.98:0/2270032962 learned_addr learned my addr 172.21.15.98:0/2270032962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8f400700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9010a590 0x7fbb9019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb87e00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 0x7fbb901a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 -- 172.21.15.98:0/2270032962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 msgr2=0x7fbb901a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 0x7fbb901a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 -- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9010a590 msgr2=0x7fbb9019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9010a590 0x7fbb9019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.501+0000 7fbb8ea00700 1 -- 172.21.15.98:0/2270032962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9019b260 con 0x7fbb901044e0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb8ea00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000ec10 tx=0x7fbb8000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000cc90 con 0x7fbb901044e0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8000ce30 con 0x7fbb901044e0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9019b550 con 0x7fbb901044e0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80010690 con 0x7fbb901044e0 2024-03-16T12:20:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.502+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb901a7810 con 0x7fbb901044e0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.503+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90068f90 con 0x7fbb901044e0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.505+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbb80003680 con 0x7fbb901044e0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.506+0000 7fbb87400700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 0x7fbb740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.507+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbb800cf280 con 0x7fbb901044e0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.507+0000 7fbb87e00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 0x7fbb740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.507+0000 7fbb87e00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 0x7fbb740a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098f0 tx=0x7fbb78019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:12.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.509+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb8009be00 con 0x7fbb901044e0 2024-03-16T12:20:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.749+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbb9002db70 con 0x7fbb901044e0 2024-03-16T12:20:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.750+0000 7fbb87400700 1 -- 172.21.15.98:0/2270032962 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbb8009b700 con 0x7fbb901044e0 2024-03-16T12:20:12.733 INFO:teuthology.orchestra.run.smithi098.stdout:197568495675 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.756+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 msgr2=0x7fbb740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.756+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 0x7fbb740a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098f0 tx=0x7fbb78019040 comp rx=0 tx=0).stop 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.756+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 msgr2=0x7fbb901a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.756+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000ec10 tx=0x7fbb8000ef20 comp rx=0 tx=0).stop 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 shutdown_connections 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9010a590 0x7fbb9019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb901044e0 0x7fbb901a0940 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fbb740a29a0 0x7fbb740a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 --2- 172.21.15.98:0/2270032962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb90104ea0 0x7fbb901a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.757+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 >> 172.21.15.98:0/2270032962 conn(0x7fbb900fffb0 msgr2=0x7fbb901016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:12.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.760+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 shutdown_connections 2024-03-16T12:20:12.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:12.760+0000 7fbb95a00700 1 -- 172.21.15.98:0/2270032962 wait complete. 2024-03-16T12:20:13.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:13 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2270032962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T12:20:13.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:13 smithi173 ceph-mon[36632]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:20:13.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:13 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2270032962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T12:20:13.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:13 smithi098 ceph-mon[31633]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:20:13.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:13 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2270032962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T12:20:13.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:13 smithi153 ceph-mon[36901]: pgmap v366: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 -- 172.21.15.98:0/560203309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 msgr2=0x7f81c410a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 --2- 172.21.15.98:0/560203309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c410a970 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f81b40099c0 tx=0x7f81b4009cd0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 -- 172.21.15.98:0/560203309 shutdown_connections 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 --2- 172.21.15.98:0/560203309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c410a970 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 --2- 172.21.15.98:0/560203309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c4104ea0 0x7f81c4111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 --2- 172.21.15.98:0/560203309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c41044e0 0x7f81c4104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 -- 172.21.15.98:0/560203309 >> 172.21.15.98:0/560203309 conn(0x7f81c40fffb0 msgr2=0x7f81c41023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 -- 172.21.15.98:0/560203309 shutdown_connections 2024-03-16T12:20:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.630+0000 7f81c9c00700 1 -- 172.21.15.98:0/560203309 wait complete. 2024-03-16T12:20:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 Processor -- start 2024-03-16T12:20:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 -- start start 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 0x7f81c41a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c4104ea0 0x7f81c41a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81c4114820 con 0x7f81c41044e0 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81c41146a0 con 0x7f81c410a590 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.631+0000 7f81c9c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f81c41149a0 con 0x7f81c4104ea0 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 0x7f81c41a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c2a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 0x7f81c41a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43492/0 (socket says 172.21.15.98:43492) 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c2a00700 1 -- 172.21.15.98:0/4276114232 learned_addr learned my addr 172.21.15.98:0/4276114232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c2000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c4104ea0 0x7f81c41a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 -- 172.21.15.98:0/4276114232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c4104ea0 msgr2=0x7f81c41a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c4104ea0 0x7f81c41a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 -- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 msgr2=0x7f81c41a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 0x7f81c41a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.632+0000 7f81c3400700 1 -- 172.21.15.98:0/4276114232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c419b140 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81c3400700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c419a9e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81bc007f10 tx=0x7f81bc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc004db0 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81bc009e20 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c419b430 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc015400 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.633+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c41a76f0 con 0x7f81c410a590 2024-03-16T12:20:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.634+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c4068f90 con 0x7f81c410a590 2024-03-16T12:20:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.636+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f81bc00db20 con 0x7f81c410a590 2024-03-16T12:20:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.637+0000 7f81bb400700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 0x7f81a80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.637+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f81bc0ce6f0 con 0x7f81c410a590 2024-03-16T12:20:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.637+0000 7f81c2a00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 0x7f81a80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.638+0000 7f81c2a00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 0x7f81a80a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009990 tx=0x7f81b4005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.641+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81bc010070 con 0x7f81c410a590 2024-03-16T12:20:13.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.890+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f81c41a7920 con 0x7f81c410a590 2024-03-16T12:20:13.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.890+0000 7f81bb400700 1 -- 172.21.15.98:0/4276114232 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f81bc09b3d0 con 0x7f81c410a590 2024-03-16T12:20:13.872 INFO:teuthology.orchestra.run.smithi098.stdout:163208757314 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 msgr2=0x7f81a80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 0x7f81a80a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009990 tx=0x7f81b4005ab0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 msgr2=0x7f81c419a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c419a9e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81bc007f10 tx=0x7f81bc00d3b0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 shutdown_connections 2024-03-16T12:20:13.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f81a80a2910 0x7f81a80a4dd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81c410a590 0x7f81c419a9e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81c41044e0 0x7f81c41a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 --2- 172.21.15.98:0/4276114232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81c4104ea0 0x7f81c41a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 >> 172.21.15.98:0/4276114232 conn(0x7f81c40fffb0 msgr2=0x7f81c4101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 shutdown_connections 2024-03-16T12:20:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:13.893+0000 7f81c9c00700 1 -- 172.21.15.98:0/4276114232 wait complete. 2024-03-16T12:20:14.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:14 smithi173 ceph-mon[36632]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-16T12:20:14.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:14 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4276114232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T12:20:14.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:14 smithi098 ceph-mon[31633]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-16T12:20:14.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:14 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4276114232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T12:20:14.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:14 smithi153 ceph-mon[36901]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-16T12:20:14.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:14 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4276114232' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T12:20:14.677 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495675 for osd.6 2024-03-16T12:20:14.677 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:15.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:15 smithi098 ceph-mon[31633]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:15.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:15 smithi173 ceph-mon[36632]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:15.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:15 smithi153 ceph-mon[36901]: pgmap v367: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:17.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:17 smithi098 ceph-mon[31633]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T12:20:18.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:17 smithi153 ceph-mon[36901]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T12:20:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:17 smithi173 ceph-mon[36632]: pgmap v368: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T12:20:20.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:19 smithi098 ceph-mon[31633]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T12:20:20.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:19 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:19 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:19 smithi153 ceph-mon[36901]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T12:20:20.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:19 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:19 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:19 smithi173 ceph-mon[36632]: pgmap v369: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T12:20:20.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:19 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:19 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 -- 172.21.15.98:0/2327294120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102580 msgr2=0x7fcbb8102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2327294120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102580 0x7fcbb8102960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009990 tx=0x7fcba8009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 -- 172.21.15.98:0/2327294120 shutdown_connections 2024-03-16T12:20:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2327294120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102580 0x7fcbb8102960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2327294120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2327294120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102f30 0x7fcbb8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 -- 172.21.15.98:0/2327294120 >> 172.21.15.98:0/2327294120 conn(0x7fcbb8079430 msgr2=0x7fcbb8079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 -- 172.21.15.98:0/2327294120 shutdown_connections 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.840+0000 7fcbbd000700 1 -- 172.21.15.98:0/2327294120 wait complete. 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.841+0000 7fcbbd000700 1 Processor -- start 2024-03-16T12:20:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.841+0000 7fcbbd000700 1 -- start start 2024-03-16T12:20:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.841+0000 7fcbbd000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 0x7fcbb80745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.841+0000 7fcbbd000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 0x7fcbb8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbbd000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb8075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbbd000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb8112680 con 0x7fcbb8107650 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbbd000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb8112500 con 0x7fcbb8102f30 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbbd000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb8112800 con 0x7fcbb8102580 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 0x7fcbb80745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 0x7fcbb80745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:47830/0 (socket says 172.21.15.98:47830) 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5e00700 1 -- 172.21.15.98:0/2782110529 learned_addr learned my addr 172.21.15.98:0/2782110529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb6800700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb8075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 0x7fcbb8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 -- 172.21.15.98:0/2782110529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 msgr2=0x7fcbb80745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 0x7fcbb80745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 -- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 msgr2=0x7fcbb8075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb8075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb5400700 1 -- 172.21.15.98:0/2782110529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb81a70a0 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.842+0000 7fcbb6800700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbb5400700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 0x7fcbb8074b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcba400e9c0 tx=0x7fcba400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba400cb70 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4004500 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb81a7390 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4010430 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.843+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb81a7920 con 0x7fcbb8102f30 2024-03-16T12:20:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.844+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb8068f90 con 0x7fcbb8102f30 2024-03-16T12:20:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.846+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcba4003eb0 con 0x7fcbb8102f30 2024-03-16T12:20:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.847+0000 7fcbaea00700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 0x7fcb9c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.848+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcba40cf060 con 0x7fcbb8102f30 2024-03-16T12:20:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.848+0000 7fcbb5e00700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 0x7fcb9c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.848+0000 7fcbb5e00700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 0x7fcb9c0a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcba8006010 tx=0x7fcba801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.856+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcba409bd40 con 0x7fcbb8102f30 2024-03-16T12:20:20.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 -- 172.21.15.98:0/3707164246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 msgr2=0x7f557010f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 --2- 172.21.15.98:0/3707164246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557010f220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f556800b0d0 tx=0x7f556800b3e0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 -- 172.21.15.98:0/3707164246 shutdown_connections 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 --2- 172.21.15.98:0/3707164246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f5570102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 --2- 172.21.15.98:0/3707164246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557010f220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 --2- 172.21.15.98:0/3707164246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f5570106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 -- 172.21.15.98:0/3707164246 >> 172.21.15.98:0/3707164246 conn(0x7f5570079430 msgr2=0x7f5570079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 -- 172.21.15.98:0/3707164246 shutdown_connections 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.750+0000 7f5575400700 1 -- 172.21.15.98:0/3707164246 wait complete. 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.752+0000 7f5575400700 1 Processor -- start 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.752+0000 7f5575400700 1 -- start start 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.752+0000 7f5575400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f5575400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f55701a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f5575400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f5575400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55701126a0 con 0x7f5570102f30 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f5575400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5570112520 con 0x7f5570102580 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f5575400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5570112820 con 0x7f5570107650 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53642/0 (socket says 172.21.15.98:53642) 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 -- 172.21.15.98:0/3114520239 learned_addr learned my addr 172.21.15.98:0/3114520239 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556f200700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556de00700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f55701a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 -- 172.21.15.98:0/3114520239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 msgr2=0x7f557019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 -- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 msgr2=0x7f55701a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f55701a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556e800700 1 -- 172.21.15.98:0/3114520239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557019b230 con 0x7f5570102580 2024-03-16T12:20:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.753+0000 7f556de00700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f55701a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f556e800700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9a0 tx=0x7f556000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556000cbd0 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556000cd70 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560018750 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557019b520 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.754+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55701a77e0 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.755+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5570068f90 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.757+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5560010430 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.758+0000 7f5567400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 0x7f55540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.758+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55600cf060 con 0x7f5570102580 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.759+0000 7f556de00700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 0x7f55540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.759+0000 7f556de00700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 0x7f55540a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5570076560 tx=0x7f555c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.762+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f556009bd40 con 0x7f5570102580 2024-03-16T12:20:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.995+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f55701a79b0 con 0x7f5570102580 2024-03-16T12:20:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.996+0000 7f5567400700 1 -- 172.21.15.98:0/3114520239 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5560097030 con 0x7f5570102580 2024-03-16T12:20:20.979 INFO:teuthology.orchestra.run.smithi098.stdout:107374182478 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 msgr2=0x7f55540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 0x7f55540a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5570076560 tx=0x7f555c009260 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 msgr2=0x7f55701a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9a0 tx=0x7f556000ecb0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 shutdown_connections 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55540a2910 0x7f55540a4dd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5570102580 0x7f55701a0910 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5570107650 0x7f557019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 --2- 172.21.15.98:0/3114520239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5570102f30 0x7f55701a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:20.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 >> 172.21.15.98:0/3114520239 conn(0x7f5570079430 msgr2=0x7f557010d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 shutdown_connections 2024-03-16T12:20:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:20.999+0000 7f5575400700 1 -- 172.21.15.98:0/3114520239 wait complete. 2024-03-16T12:20:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:20:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:20:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.089+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcbb8075b00 con 0x7fcbb8102f30 2024-03-16T12:20:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.090+0000 7fcbaea00700 1 -- 172.21.15.98:0/2782110529 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcba4097030 con 0x7fcbb8102f30 2024-03-16T12:20:21.072 INFO:teuthology.orchestra.run.smithi098.stdout:55834574938 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.092+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 msgr2=0x7fcb9c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.092+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 0x7fcb9c0a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcba8006010 tx=0x7fcba801a040 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.092+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 msgr2=0x7fcbb8074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.092+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 0x7fcbb8074b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcba400e9c0 tx=0x7fcba400ecd0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 shutdown_connections 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbb8107650 0x7fcbb8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbb8102580 0x7fcbb80745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb9c0a2910 0x7fcb9c0a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 --2- 172.21.15.98:0/2782110529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb8102f30 0x7fcbb8074b10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 >> 172.21.15.98:0/2782110529 conn(0x7fcbb8079430 msgr2=0x7fcbb8101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 shutdown_connections 2024-03-16T12:20:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.093+0000 7fcbbd000700 1 -- 172.21.15.98:0/2782110529 wait complete. 2024-03-16T12:20:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:21.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:20:21.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:20:21.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:21.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:20:21.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:20:21.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:20:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 -- 172.21.15.98:0/3189880636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac108320 msgr2=0x7f33ac108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 --2- 172.21.15.98:0/3189880636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac108320 0x7f33ac108700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009990 tx=0x7f33a0009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 -- 172.21.15.98:0/3189880636 shutdown_connections 2024-03-16T12:20:21.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 --2- 172.21.15.98:0/3189880636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac108320 0x7f33ac108700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 --2- 172.21.15.98:0/3189880636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac06a920 0x7f33ac10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 --2- 172.21.15.98:0/3189880636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 0x7f33ac06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 -- 172.21.15.98:0/3189880636 >> 172.21.15.98:0/3189880636 conn(0x7f33ac078100 msgr2=0x7f33ac078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 -- 172.21.15.98:0/3189880636 shutdown_connections 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.728+0000 7f33b3600700 1 -- 172.21.15.98:0/3189880636 wait complete. 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.729+0000 7f33b3600700 1 Processor -- start 2024-03-16T12:20:21.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.729+0000 7f33b3600700 1 -- start start 2024-03-16T12:20:21.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 0x7f33ac19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:21.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac06a920 0x7f33ac19b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:21.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1125b0 con 0x7f33ac069f80 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac112430 con 0x7f33ac06a920 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b3600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac112730 con 0x7f33ac108320 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b1600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b0c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 0x7f33ac19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b1600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:47846/0 (socket says 172.21.15.98:47846) 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33b1600700 1 -- 172.21.15.98:0/1468094836 learned_addr learned my addr 172.21.15.98:0/1468094836 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:21.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.730+0000 7f33abe00700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac06a920 0x7f33ac19b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 -- 172.21.15.98:0/1468094836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac06a920 msgr2=0x7f33ac19b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac06a920 0x7f33ac19b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 -- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 msgr2=0x7f33ac19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 0x7f33ac19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 -- 172.21.15.98:0/1468094836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac19fd20 con 0x7f33ac108320 2024-03-16T12:20:21.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.731+0000 7f33b1600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f33a4007ee0 tx=0x7f33a400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.732+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400dd20 con 0x7f33ac108320 2024-03-16T12:20:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.732+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a4004020 con 0x7f33ac108320 2024-03-16T12:20:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.732+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4015660 con 0x7f33ac108320 2024-03-16T12:20:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.732+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac1a0010 con 0x7f33ac108320 2024-03-16T12:20:21.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.732+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac1a7890 con 0x7f33ac108320 2024-03-16T12:20:21.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.733+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33ac068f90 con 0x7f33ac108320 2024-03-16T12:20:21.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.735+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f33a40041d0 con 0x7f33ac108320 2024-03-16T12:20:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.736+0000 7f33a9600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 0x7f33940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.736+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f33a40ce750 con 0x7f33ac108320 2024-03-16T12:20:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.736+0000 7f33b0c00700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 0x7f33940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:21.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.737+0000 7f33b0c00700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 0x7f33940a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f33a000b2e0 tx=0x7f33a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:21.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.740+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f33a4010070 con 0x7f33ac108320 2024-03-16T12:20:21.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.972+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f33ac1a7ac0 con 0x7f33ac108320 2024-03-16T12:20:21.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.972+0000 7f33a9600700 1 -- 172.21.15.98:0/1468094836 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f33a409b430 con 0x7f33ac108320 2024-03-16T12:20:21.954 INFO:teuthology.orchestra.run.smithi098.stdout:343597383715 2024-03-16T12:20:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 msgr2=0x7f33940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 0x7f33940a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f33a000b2e0 tx=0x7f33a0005fd0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 msgr2=0x7f33ac19f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f33a4007ee0 tx=0x7f33a400d3b0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 shutdown_connections 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33ac108320 0x7f33ac19f5e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33ac069f80 0x7f33ac19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f33940a2910 0x7f33940a4dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 --2- 172.21.15.98:0/1468094836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33ac06a920 0x7f33ac19b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.975+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 >> 172.21.15.98:0/1468094836 conn(0x7f33ac078100 msgr2=0x7f33ac10d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.976+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 shutdown_connections 2024-03-16T12:20:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:21.976+0000 7f33b3600700 1 -- 172.21.15.98:0/1468094836 wait complete. 2024-03-16T12:20:21.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:21 smithi098 ceph-mon[31633]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:21.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:21 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3114520239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T12:20:21.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:21 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2782110529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T12:20:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:21 smithi153 ceph-mon[36901]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:21 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3114520239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T12:20:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:21 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2782110529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T12:20:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:21 smithi173 ceph-mon[36632]: pgmap v370: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:21 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3114520239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T12:20:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:21 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2782110529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T12:20:23.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:22 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1468094836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T12:20:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:22 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1468094836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T12:20:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:22 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1468094836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 -- 172.21.15.98:0/320813850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9401044e0 msgr2=0x7fe940104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 --2- 172.21.15.98:0/320813850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9401044e0 0x7fe940104960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe9280098f0 tx=0x7fe928009c00 comp rx=0 tx=0).stop 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 -- 172.21.15.98:0/320813850 shutdown_connections 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 --2- 172.21.15.98:0/320813850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe94010a590 0x7fe94010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 --2- 172.21.15.98:0/320813850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe940111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 --2- 172.21.15.98:0/320813850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9401044e0 0x7fe940104960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.721+0000 7fe947200700 1 -- 172.21.15.98:0/320813850 >> 172.21.15.98:0/320813850 conn(0x7fe9400fffb0 msgr2=0x7fe9401023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.722+0000 7fe947200700 1 -- 172.21.15.98:0/320813850 shutdown_connections 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.722+0000 7fe947200700 1 -- 172.21.15.98:0/320813850 wait complete. 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 Processor -- start 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 -- start start 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9401044e0 0x7fe9401a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 0x7fe94019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe940114830 con 0x7fe940104ea0 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401146b0 con 0x7fe9401044e0 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.723+0000 7fe947200700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9401149b0 con 0x7fe94010a590 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe93fe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9401044e0 0x7fe9401a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe937400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 0x7fe94019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe937400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60454/0 (socket says 172.21.15.98:60454) 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe937400700 1 -- 172.21.15.98:0/3636615736 learned_addr learned my addr 172.21.15.98:0/3636615736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 -- 172.21.15.98:0/3636615736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9401044e0 msgr2=0x7fe9401a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9401044e0 0x7fe9401a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 -- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 msgr2=0x7fe9401a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 -- 172.21.15.98:0/3636615736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94019b210 con 0x7fe94010a590 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe937400700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe945200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 0x7fe94019aab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe93800d750 tx=0x7fe93800da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.724+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938014030 con 0x7fe94010a590 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.725+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe938004500 con 0x7fe94010a590 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.725+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94019b500 con 0x7fe94010a590 2024-03-16T12:20:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.725+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938005490 con 0x7fe94010a590 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.725+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9401a7770 con 0x7fe94010a590 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.726+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94010ca20 con 0x7fe94010a590 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.727+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe938004750 con 0x7fe94010a590 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.729+0000 7fe93d600700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 0x7fe9240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.729+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe9380ce6d0 con 0x7fe94010a590 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.729+0000 7fe93fe00700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 0x7fe9240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.730+0000 7fe93fe00700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 0x7fe9240a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe9401095d0 tx=0x7fe93000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.732+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe93809b3b0 con 0x7fe94010a590 2024-03-16T12:20:23.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.959+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe94019bf90 con 0x7fe94010a590 2024-03-16T12:20:23.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.959+0000 7fe93d600700 1 -- 172.21.15.98:0/3636615736 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe93809acb0 con 0x7fe94010a590 2024-03-16T12:20:23.941 INFO:teuthology.orchestra.run.smithi098.stdout:219043332153 2024-03-16T12:20:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 msgr2=0x7fe9240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 0x7fe9240a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe9401095d0 tx=0x7fe93000b040 comp rx=0 tx=0).stop 2024-03-16T12:20:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 msgr2=0x7fe94019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 0x7fe94019aab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe93800d750 tx=0x7fe93800da60 comp rx=0 tx=0).stop 2024-03-16T12:20:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 shutdown_connections 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe9240a2890 0x7fe9240a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9401044e0 0x7fe9401a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94010a590 0x7fe94019aab0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 --2- 172.21.15.98:0/3636615736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe940104ea0 0x7fe9401a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.962+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 >> 172.21.15.98:0/3636615736 conn(0x7fe9400fffb0 msgr2=0x7fe940101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.963+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 shutdown_connections 2024-03-16T12:20:23.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:23.963+0000 7fe947200700 1 -- 172.21.15.98:0/3636615736 wait complete. 2024-03-16T12:20:24.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:23 smithi098 ceph-mon[31633]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:24.209 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383709 got 343597383715 for osd.11 2024-03-16T12:20:24.210 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:23 smithi153 ceph-mon[36901]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:23 smithi173 ceph-mon[36632]: pgmap v371: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:24.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574938 for osd.1 2024-03-16T12:20:24.878 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:25.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:24 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3636615736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T12:20:25.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:24 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3636615736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T12:20:25.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:24 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3636615736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T12:20:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 -- 172.21.15.98:0/390504743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688102320 msgr2=0x7fb6881027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 --2- 172.21.15.98:0/390504743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688102320 0x7fb6881027a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb6700099c0 tx=0x7fb670009cd0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 -- 172.21.15.98:0/390504743 shutdown_connections 2024-03-16T12:20:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 --2- 172.21.15.98:0/390504743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102ce0 0x7fb68810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 --2- 172.21.15.98:0/390504743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688108300 0x7fb6881086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 --2- 172.21.15.98:0/390504743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688102320 0x7fb6881027a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 -- 172.21.15.98:0/390504743 >> 172.21.15.98:0/390504743 conn(0x7fb688079400 msgr2=0x7fb688079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:25.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 -- 172.21.15.98:0/390504743 shutdown_connections 2024-03-16T12:20:25.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.421+0000 7fb68e000700 1 -- 172.21.15.98:0/390504743 wait complete. 2024-03-16T12:20:25.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.422+0000 7fb68e000700 1 Processor -- start 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.422+0000 7fb68e000700 1 -- start start 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 0x7fb6881a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 0x7fb6881a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688108300 0x7fb68819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6881125d0 con 0x7fb688102320 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb688112450 con 0x7fb688108300 2024-03-16T12:20:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb68e000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb688112750 con 0x7fb688102ce0 2024-03-16T12:20:25.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb687400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 0x7fb6881a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:25.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb67fe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 0x7fb6881a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:25.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb687400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 0x7fb6881a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60474/0 (socket says 172.21.15.98:60474) 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb687400700 1 -- 172.21.15.98:0/3182793776 learned_addr learned my addr 172.21.15.98:0/3182793776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.423+0000 7fb687e00700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688108300 0x7fb68819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 -- 172.21.15.98:0/3182793776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688108300 msgr2=0x7fb68819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688108300 0x7fb68819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 -- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 msgr2=0x7fb6881a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 0x7fb6881a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 -- 172.21.15.98:0/3182793776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68819b170 con 0x7fb688102ce0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb67fe00700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 0x7fb6881a0d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6700098e0 tx=0x7fb670003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670003680 con 0x7fb688102ce0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6700055a0 con 0x7fb688102ce0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67000fc90 con 0x7fb688102ce0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68819b400 con 0x7fb688102ce0 2024-03-16T12:20:25.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.424+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6881a7720 con 0x7fb688102ce0 2024-03-16T12:20:25.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.425+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6881039b0 con 0x7fb688102ce0 2024-03-16T12:20:25.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.427+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb67000f460 con 0x7fb688102ce0 2024-03-16T12:20:25.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.428+0000 7fb684c00700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 0x7fb66c0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:25.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.429+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb6700cfba0 con 0x7fb688102ce0 2024-03-16T12:20:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.429+0000 7fb687400700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 0x7fb66c0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.429+0000 7fb687400700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 0x7fb66c0a4ef0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb678009da0 tx=0x7fb67800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.431+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6700cfe20 con 0x7fb688102ce0 2024-03-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.652+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb6881a78f0 con 0x7fb688102ce0 2024-03-16T12:20:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.653+0000 7fb684c00700 1 -- 172.21.15.98:0/3182793776 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb670013070 con 0x7fb688102ce0 2024-03-16T12:20:25.635 INFO:teuthology.orchestra.run.smithi098.stdout:141733920844 2024-03-16T12:20:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 msgr2=0x7fb66c0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 0x7fb66c0a4ef0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb678009da0 tx=0x7fb67800b040 comp rx=0 tx=0).stop 2024-03-16T12:20:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 msgr2=0x7fb6881a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 0x7fb6881a0d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6700098e0 tx=0x7fb670003a90 comp rx=0 tx=0).stop 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 shutdown_connections 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb66c0a2a30 0x7fb66c0a4ef0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb688108300 0x7fb68819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb688102320 0x7fb6881a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 --2- 172.21.15.98:0/3182793776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb688102ce0 0x7fb6881a0d90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 >> 172.21.15.98:0/3182793776 conn(0x7fb688079400 msgr2=0x7fb688074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 shutdown_connections 2024-03-16T12:20:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:25.656+0000 7fb68e000700 1 -- 172.21.15.98:0/3182793776 wait complete. 2024-03-16T12:20:26.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:25 smithi098 ceph-mon[31633]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:26.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:25 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3182793776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T12:20:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:25 smithi153 ceph-mon[36901]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:26.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:25 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3182793776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T12:20:26.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:25 smithi173 ceph-mon[36632]: pgmap v372: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:26.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:25 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3182793776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T12:20:26.462 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182478 for osd.3 2024-03-16T12:20:26.462 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:27.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757308 got 163208757314 for osd.5 2024-03-16T12:20:27.548 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:28.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:27 smithi098 ceph-mon[31633]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T12:20:28.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332147 got 219043332153 for osd.7 2024-03-16T12:20:28.113 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:28.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:27 smithi153 ceph-mon[36901]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T12:20:28.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:27 smithi173 ceph-mon[36632]: pgmap v373: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T12:20:28.610 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920836 got 141733920844 for osd.4 2024-03-16T12:20:28.611 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 -- 172.21.15.98:0/570784534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 msgr2=0x7f8b1806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/570784534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b1806a3e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8b040099c0 tx=0x7f8b04009cd0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 -- 172.21.15.98:0/570784534 shutdown_connections 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/570784534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b1806a920 0x7f8b1810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/570784534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b18108320 0x7f8b18108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/570784534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b1806a3e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 -- 172.21.15.98:0/570784534 >> 172.21.15.98:0/570784534 conn(0x7f8b18078100 msgr2=0x7f8b18078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.280+0000 7f8b1fe00700 1 -- 172.21.15.98:0/570784534 shutdown_connections 2024-03-16T12:20:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.281+0000 7f8b1fe00700 1 -- 172.21.15.98:0/570784534 wait complete. 2024-03-16T12:20:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.282+0000 7f8b1fe00700 1 Processor -- start 2024-03-16T12:20:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.282+0000 7f8b1fe00700 1 -- start start 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b181012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b1806a920 0x7f8b181017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 0x7f8b18101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181125d0 con 0x7f8b18069f80 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b18112450 con 0x7f8b18108320 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1fe00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b18112750 con 0x7f8b1806a920 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b181012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1ca00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b1806a920 0x7f8b181017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1d400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b181012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43488/0 (socket says 172.21.15.98:43488) 2024-03-16T12:20:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 0x7f8b18101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1d400700 1 -- 172.21.15.98:0/4067640497 learned_addr learned my addr 172.21.15.98:0/4067640497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 -- 172.21.15.98:0/4067640497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b1806a920 msgr2=0x7f8b181017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b1806a920 0x7f8b181017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 -- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 msgr2=0x7f8b181012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b181012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.283+0000 7f8b1de00700 1 -- 172.21.15.98:0/4067640497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b181a6fd0 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b1de00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 0x7f8b18101d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e970 tx=0x7f8b1000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10004220 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b10004d10 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b181a72c0 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10010700 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.284+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b181a77d0 con 0x7f8b18108320 2024-03-16T12:20:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.285+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b18068f90 con 0x7f8b18108320 2024-03-16T12:20:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.287+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8b10004750 con 0x7f8b18108320 2024-03-16T12:20:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.288+0000 7f8b16000700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 0x7f8b000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.289+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8b100ceb30 con 0x7f8b18108320 2024-03-16T12:20:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.289+0000 7f8b1d400700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 0x7f8b000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.289+0000 7f8b1d400700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 0x7f8b000a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009c10 tx=0x7f8b0c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.292+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b1001e300 con 0x7f8b18108320 2024-03-16T12:20:29.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.519+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8b1802db70 con 0x7f8b18108320 2024-03-16T12:20:29.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.520+0000 7f8b16000700 1 -- 172.21.15.98:0/4067640497 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8b1009b810 con 0x7f8b18108320 2024-03-16T12:20:29.502 INFO:teuthology.orchestra.run.smithi098.stdout:287762808878 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 msgr2=0x7f8b000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 0x7f8b000a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009c10 tx=0x7f8b0c00b040 comp rx=0 tx=0).stop 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 msgr2=0x7f8b18101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 0x7f8b18101d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e970 tx=0x7f8b1000ec80 comp rx=0 tx=0).stop 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 shutdown_connections 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8b000a2910 0x7f8b000a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b18108320 0x7f8b18101d20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b18069f80 0x7f8b181012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 --2- 172.21.15.98:0/4067640497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b1806a920 0x7f8b181017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 >> 172.21.15.98:0/4067640497 conn(0x7f8b18078100 msgr2=0x7f8b180ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 shutdown_connections 2024-03-16T12:20:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:29.524+0000 7f8b1fe00700 1 -- 172.21.15.98:0/4067640497 wait complete. 2024-03-16T12:20:30.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:29 smithi098 ceph-mon[31633]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:30.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4067640497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T12:20:30.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:29 smithi153 ceph-mon[36901]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:30.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4067640497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T12:20:30.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:29 smithi173 ceph-mon[36632]: pgmap v374: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T12:20:30.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4067640497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T12:20:31.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808878 for osd.9 2024-03-16T12:20:31.256 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 -- 172.21.15.98:0/1810331462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 msgr2=0x7fc68c10e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 --2- 172.21.15.98:0/1810331462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c10e790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc674009990 tx=0x7fc674009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 -- 172.21.15.98:0/1810331462 shutdown_connections 2024-03-16T12:20:31.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 --2- 172.21.15.98:0/1810331462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 0x7fc68c1110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 --2- 172.21.15.98:0/1810331462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c0748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 --2- 172.21.15.98:0/1810331462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c10e790 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 -- 172.21.15.98:0/1810331462 >> 172.21.15.98:0/1810331462 conn(0x7fc68c079400 msgr2=0x7fc68c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 -- 172.21.15.98:0/1810331462 shutdown_connections 2024-03-16T12:20:31.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.864+0000 7fc692400700 1 -- 172.21.15.98:0/1810331462 wait complete. 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.865+0000 7fc692400700 1 Processor -- start 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.865+0000 7fc692400700 1 -- start start 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 0x7fc68c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1147d0 con 0x7fc68c10ecd0 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c114650 con 0x7fc68c074e20 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc692400700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c114950 con 0x7fc68c074500 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc68b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc682a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:31.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc68be00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 0x7fc68c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:31.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc682a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:57924/0 (socket says 172.21.15.98:57924) 2024-03-16T12:20:31.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc68b400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59690/0 (socket says 172.21.15.98:59690) 2024-03-16T12:20:31.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.866+0000 7fc682a00700 1 -- 172.21.15.98:0/623276473 learned_addr learned my addr 172.21.15.98:0/623276473 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 -- 172.21.15.98:0/623276473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 msgr2=0x7fc68c1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 -- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 msgr2=0x7fc68c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 0x7fc68c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 -- 172.21.15.98:0/623276473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c19b230 con 0x7fc68c074e20 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc682a00700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc674000c00 tx=0x7fc67400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674027070 con 0x7fc68c074e20 2024-03-16T12:20:31.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6740056d0 con 0x7fc68c074e20 2024-03-16T12:20:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68c19b4c0 con 0x7fc68c074e20 2024-03-16T12:20:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674022e10 con 0x7fc68c074e20 2024-03-16T12:20:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.867+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68c1a7790 con 0x7fc68c074e20 2024-03-16T12:20:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.868+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc68c068f90 con 0x7fc68c074e20 2024-03-16T12:20:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.870+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc67402a070 con 0x7fc68c074e20 2024-03-16T12:20:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.872+0000 7fc688c00700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 0x7fc6700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.872+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc6740cfa40 con 0x7fc68c074e20 2024-03-16T12:20:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.872+0000 7fc68b400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 0x7fc6700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.873+0000 7fc68b400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 0x7fc6700a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc68c079c10 tx=0x7fc67c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:31.875+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc674013070 con 0x7fc68c074e20 2024-03-16T12:20:32.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.102+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fc68c02db70 con 0x7fc68c074e20 2024-03-16T12:20:32.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:31 smithi098 ceph-mon[31633]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:32.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.103+0000 7fc688c00700 1 -- 172.21.15.98:0/623276473 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fc67409c7b0 con 0x7fc68c074e20 2024-03-16T12:20:32.085 INFO:teuthology.orchestra.run.smithi098.stdout:309237645352 2024-03-16T12:20:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.105+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 msgr2=0x7fc6700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.105+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 0x7fc6700a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc68c079c10 tx=0x7fc67c009260 comp rx=0 tx=0).stop 2024-03-16T12:20:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.105+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 msgr2=0x7fc68c1a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.105+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc674000c00 tx=0x7fc67400bab0 comp rx=0 tx=0).stop 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 shutdown_connections 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc68c10ecd0 0x7fc68c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc68c074500 0x7fc68c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6700a2910 0x7fc6700a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 --2- 172.21.15.98:0/623276473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc68c074e20 0x7fc68c1a0e00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 >> 172.21.15.98:0/623276473 conn(0x7fc68c079400 msgr2=0x7fc68c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 shutdown_connections 2024-03-16T12:20:32.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:32.106+0000 7fc692400700 1 -- 172.21.15.98:0/623276473 wait complete. 2024-03-16T12:20:32.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:31 smithi153 ceph-mon[36901]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:32.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:31 smithi173 ceph-mon[36632]: pgmap v375: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:32.903 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645346 got 309237645352 for osd.10 2024-03-16T12:20:32.903 DEBUG:teuthology.parallel:result is None 2024-03-16T12:20:32.904 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T12:20:32.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T12:20:33.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:32 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/623276473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T12:20:33.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:32 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/623276473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T12:20:33.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:32 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/623276473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T12:20:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:34.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:33 smithi098 ceph-mon[31633]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:34.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:33 smithi153 ceph-mon[36901]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:34.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:33 smithi173 ceph-mon[36632]: pgmap v376: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 -- 172.21.15.98:0/764981210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08102580 msgr2=0x7f9f08102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/764981210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08102580 0x7f9f08102960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009990 tx=0x7f9ef0009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 -- 172.21.15.98:0/764981210 shutdown_connections 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/764981210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08102580 0x7f9f08102960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/764981210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08107650 0x7f9f0810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/764981210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f08106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 -- 172.21.15.98:0/764981210 >> 172.21.15.98:0/764981210 conn(0x7f9f08079430 msgr2=0x7f9f08079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 -- 172.21.15.98:0/764981210 shutdown_connections 2024-03-16T12:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.575+0000 7f9f0fa00700 1 -- 172.21.15.98:0/764981210 wait complete. 2024-03-16T12:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.576+0000 7f9f0fa00700 1 Processor -- start 2024-03-16T12:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 -- start start 2024-03-16T12:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08102580 0x7f9f0819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08107650 0x7f9f0819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f081126a0 con 0x7f9f08102580 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f08112520 con 0x7f9f08107650 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0fa00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9f08112820 con 0x7f9f08102f30 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9effe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0d000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08102580 0x7f9f0819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9effe00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59706/0 (socket says 172.21.15.98:59706) 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9effe00700 1 -- 172.21.15.98:0/2143069274 learned_addr learned my addr 172.21.15.98:0/2143069274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.577+0000 7f9f0da00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08107650 0x7f9f0819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 -- 172.21.15.98:0/2143069274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08107650 msgr2=0x7f9f0819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08107650 0x7f9f0819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 -- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08102580 msgr2=0x7f9f0819ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08102580 0x7f9f0819ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 -- 172.21.15.98:0/2143069274 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0819fd80 con 0x7f9f08102f30 2024-03-16T12:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9effe00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800e9f0 tx=0x7f9ef800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004a30 con 0x7f9f08102f30 2024-03-16T12:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef800cca0 con 0x7f9f08102f30 2024-03-16T12:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f081a0070 con 0x7f9f08102f30 2024-03-16T12:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.578+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80056c0 con 0x7f9f08102f30 2024-03-16T12:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.579+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f081a78f0 con 0x7f9f08102f30 2024-03-16T12:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.579+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f08068f90 con 0x7f9f08102f30 2024-03-16T12:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.581+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ef800c7c0 con 0x7f9f08102f30 2024-03-16T12:20:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.583+0000 7f9f06000700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 0x7f9ee80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.583+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9ef80cef70 con 0x7f9f08102f30 2024-03-16T12:20:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.583+0000 7f9f0d000700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 0x7f9ee80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.584+0000 7f9f0d000700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 0x7f9ee80a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0006010 tx=0x7f9ef000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.588+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ef809bc50 con 0x7f9f08102f30 2024-03-16T12:20:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.803+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f0810e590 con 0x7f9ee80a2910 2024-03-16T12:20:35.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.815+0000 7f9f06000700 1 -- 172.21.15.98:0/2143069274 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340483 (secure 0 0 0) 0x7f9f0810e590 con 0x7f9ee80a2910 2024-03-16T12:20:35.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 msgr2=0x7f9ee80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 0x7f9ee80a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0006010 tx=0x7f9ef000b260 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 msgr2=0x7f9f0819b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800e9f0 tx=0x7f9ef800ed00 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 shutdown_connections 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9ee80a2910 0x7f9ee80a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f08107650 0x7f9f0819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f08102580 0x7f9f0819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 --2- 172.21.15.98:0/2143069274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08102f30 0x7f9f0819b2b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 >> 172.21.15.98:0/2143069274 conn(0x7f9f08079430 msgr2=0x7f9f08101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.822+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 shutdown_connections 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:35.823+0000 7f9f0fa00700 1 -- 172.21.15.98:0/2143069274 wait complete. 2024-03-16T12:20:35.807 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:20:36.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:35 smithi098 ceph-mon[31633]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:36.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:35 smithi153 ceph-mon[36901]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:36.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:35 smithi173 ceph-mon[36632]: pgmap v377: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:36.859 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-03-16T12:20:34.823606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656794,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":667,"num_read_kb":447,"num_write":352,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384048,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557776,"statfs":{"total":1151940427776,"available":1151547162624,"internally_reserved":0,"allocated":14434304,"data_stored":7956092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006070"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803695+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:20:12.803695+0000","last_peered":"2024-03-16T12:20:12.803695+0000","last_clean":"2024-03-16T12:20:12.803695+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:20:12.803695+0000","last_undegraded":"2024-03-16T12:20:12.803695+0000","last_fullsized":"2024-03-16T12:20:12.803695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798064+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:20:12.798064+0000","last_peered":"2024-03-16T12:20:12.798064+0000","last_clean":"2024-03-16T12:20:12.798064+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:20:12.798064+0000","last_undegraded":"2024-03-16T12:20:12.798064+0000","last_fullsized":"2024-03-16T12:20:12.798064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799952+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:20:12.799952+0000","last_peered":"2024-03-16T12:20:12.799952+0000","last_clean":"2024-03-16T12:20:12.799952+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:20:12.799952+0000","last_undegraded":"2024-03-16T12:20:12.799952+0000","last_fullsized":"2024-03-16T12:20:12.799952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755621+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:20:12.755621+0000","last_peered":"2024-03-16T12:20:12.755621+0000","last_clean":"2024-03-16T12:20:12.755621+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:20:12.755621+0000","last_undegraded":"2024-03-16T12:20:12.755621+0000","last_fullsized":"2024-03-16T12:20:12.755621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.778564+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:20:12.778564+0000","last_peered":"2024-03-16T12:20:12.778564+0000","last_clean":"2024-03-16T12:20:12.778564+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:20:12.778564+0000","last_undegraded":"2024-03-16T12:20:12.778564+0000","last_fullsized":"2024-03-16T12:20:12.778564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785812+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:20:12.785812+0000","last_peered":"2024-03-16T12:20:12.785812+0000","last_clean":"2024-03-16T12:20:12.785812+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:20:12.785812+0000","last_undegraded":"2024-03-16T12:20:12.785812+0000","last_fullsized":"2024-03-16T12:20:12.785812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.766436+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:20:12.766436+0000","last_peered":"2024-03-16T12:20:12.766436+0000","last_clean":"2024-03-16T12:20:12.766436+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:20:12.766436+0000","last_undegraded":"2024-03-16T12:20:12.766436+0000","last_fullsized":"2024-03-16T12:20:12.766436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793048+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:20:12.793048+0000","last_peered":"2024-03-16T12:20:12.793048+0000","last_clean":"2024-03-16T12:20:12.793048+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:20:12.793048+0000","last_undegraded":"2024-03-16T12:20:12.793048+0000","last_fullsized":"2024-03-16T12:20:12.793048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782148+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:20:12.782148+0000","last_peered":"2024-03-16T12:20:12.782148+0000","last_clean":"2024-03-16T12:20:12.782148+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:20:12.782148+0000","last_undegraded":"2024-03-16T12:20:12.782148+0000","last_fullsized":"2024-03-16T12:20:12.782148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785267+0000","last_change":"2024-03-16T12:19:44.848697+0000","last_active":"2024-03-16T12:20:12.785267+0000","last_peered":"2024-03-16T12:20:12.785267+0000","last_clean":"2024-03-16T12:20:12.785267+0000","last_became_active":"2024-03-16T12:19:44.848414+0000","last_became_peered":"2024-03-16T12:19:44.848414+0000","last_unstale":"2024-03-16T12:20:12.785267+0000","last_undegraded":"2024-03-16T12:20:12.785267+0000","last_fullsized":"2024-03-16T12:20:12.785267+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:54.721056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786301+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:20:12.786301+0000","last_peered":"2024-03-16T12:20:12.786301+0000","last_clean":"2024-03-16T12:20:12.786301+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:20:12.786301+0000","last_undegraded":"2024-03-16T12:20:12.786301+0000","last_fullsized":"2024-03-16T12:20:12.786301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.754844+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:20:12.754844+0000","last_peered":"2024-03-16T12:20:12.754844+0000","last_clean":"2024-03-16T12:20:12.754844+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:20:12.754844+0000","last_undegraded":"2024-03-16T12:20:12.754844+0000","last_fullsized":"2024-03-16T12:20:12.754844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790030+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:20:12.790030+0000","last_peered":"2024-03-16T12:20:12.790030+0000","last_clean":"2024-03-16T12:20:12.790030+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:20:12.790030+0000","last_undegraded":"2024-03-16T12:20:12.790030+0000","last_fullsized":"2024-03-16T12:20:12.790030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793934+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:20:12.793934+0000","last_peered":"2024-03-16T12:20:12.793934+0000","last_clean":"2024-03-16T12:20:12.793934+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:20:12.793934+0000","last_undegraded":"2024-03-16T12:20:12.793934+0000","last_fullsized":"2024-03-16T12:20:12.793934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.795833+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:20:12.795833+0000","last_peered":"2024-03-16T12:20:12.795833+0000","last_clean":"2024-03-16T12:20:12.795833+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:20:12.795833+0000","last_undegraded":"2024-03-16T12:20:12.795833+0000","last_fullsized":"2024-03-16T12:20:12.795833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803822+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:20:12.803822+0000","last_peered":"2024-03-16T12:20:12.803822+0000","last_clean":"2024-03-16T12:20:12.803822+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:20:12.803822+0000","last_undegraded":"2024-03-16T12:20:12.803822+0000","last_fullsized":"2024-03-16T12:20:12.803822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803923+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:20:12.803923+0000","last_peered":"2024-03-16T12:20:12.803923+0000","last_clean":"2024-03-16T12:20:12.803923+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:20:12.803923+0000","last_undegraded":"2024-03-16T12:20:12.803923+0000","last_fullsized":"2024-03-16T12:20:12.803923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786099+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:20:12.786099+0000","last_peered":"2024-03-16T12:20:12.786099+0000","last_clean":"2024-03-16T12:20:12.786099+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:20:12.786099+0000","last_undegraded":"2024-03-16T12:20:12.786099+0000","last_fullsized":"2024-03-16T12:20:12.786099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792800+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:20:12.792800+0000","last_peered":"2024-03-16T12:20:12.792800+0000","last_clean":"2024-03-16T12:20:12.792800+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:20:12.792800+0000","last_undegraded":"2024-03-16T12:20:12.792800+0000","last_fullsized":"2024-03-16T12:20:12.792800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547227+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:19:44.547227+0000","last_peered":"2024-03-16T12:19:44.547227+0000","last_clean":"2024-03-16T12:19:44.547227+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:19:44.547227+0000","last_undegraded":"2024-03-16T12:19:44.547227+0000","last_fullsized":"2024-03-16T12:19:44.547227+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847242+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:44.847242+0000","last_peered":"2024-03-16T12:19:44.847242+0000","last_clean":"2024-03-16T12:19:44.847242+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:44.847242+0000","last_undegraded":"2024-03-16T12:19:44.847242+0000","last_fullsized":"2024-03-16T12:19:44.847242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.767386+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:20:12.767386+0000","last_peered":"2024-03-16T12:20:12.767386+0000","last_clean":"2024-03-16T12:20:12.767386+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:20:12.767386+0000","last_undegraded":"2024-03-16T12:20:12.767386+0000","last_fullsized":"2024-03-16T12:20:12.767386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559192+0000","last_change":"2024-03-16T12:19:44.559192+0000","last_active":"2024-03-16T12:19:44.559192+0000","last_peered":"2024-03-16T12:19:44.559192+0000","last_clean":"2024-03-16T12:19:44.559192+0000","last_became_active":"2024-03-16T12:19:44.558795+0000","last_became_peered":"2024-03-16T12:19:44.558795+0000","last_unstale":"2024-03-16T12:19:44.559192+0000","last_undegraded":"2024-03-16T12:19:44.559192+0000","last_fullsized":"2024-03-16T12:19:44.559192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:41:28.267737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550779+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:19:44.550779+0000","last_peered":"2024-03-16T12:19:44.550779+0000","last_clean":"2024-03-16T12:19:44.550779+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:19:44.550779+0000","last_undegraded":"2024-03-16T12:19:44.550779+0000","last_fullsized":"2024-03-16T12:19:44.550779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803963+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:20:12.803963+0000","last_peered":"2024-03-16T12:20:12.803963+0000","last_clean":"2024-03-16T12:20:12.803963+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:20:12.803963+0000","last_undegraded":"2024-03-16T12:20:12.803963+0000","last_fullsized":"2024-03-16T12:20:12.803963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546335+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:19:44.546335+0000","last_peered":"2024-03-16T12:19:44.546335+0000","last_clean":"2024-03-16T12:19:44.546335+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:19:44.546335+0000","last_undegraded":"2024-03-16T12:19:44.546335+0000","last_fullsized":"2024-03-16T12:19:44.546335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543804+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:44.543804+0000","last_peered":"2024-03-16T12:19:44.543804+0000","last_clean":"2024-03-16T12:19:44.543804+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:44.543804+0000","last_undegraded":"2024-03-16T12:19:44.543804+0000","last_fullsized":"2024-03-16T12:19:44.543804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788655+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:20:12.788655+0000","last_peered":"2024-03-16T12:20:12.788655+0000","last_clean":"2024-03-16T12:20:12.788655+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:20:12.788655+0000","last_undegraded":"2024-03-16T12:20:12.788655+0000","last_fullsized":"2024-03-16T12:20:12.788655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545946+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:19:44.545946+0000","last_peered":"2024-03-16T12:19:44.545946+0000","last_clean":"2024-03-16T12:19:44.545946+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:19:44.545946+0000","last_undegraded":"2024-03-16T12:19:44.545946+0000","last_fullsized":"2024-03-16T12:19:44.545946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543884+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:19:44.543884+0000","last_peered":"2024-03-16T12:19:44.543884+0000","last_clean":"2024-03-16T12:19:44.543884+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:19:44.543884+0000","last_undegraded":"2024-03-16T12:19:44.543884+0000","last_fullsized":"2024-03-16T12:19:44.543884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.784747+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:20:12.784747+0000","last_peered":"2024-03-16T12:20:12.784747+0000","last_clean":"2024-03-16T12:20:12.784747+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:20:12.784747+0000","last_undegraded":"2024-03-16T12:20:12.784747+0000","last_fullsized":"2024-03-16T12:20:12.784747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547745+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:19:44.547745+0000","last_peered":"2024-03-16T12:19:44.547745+0000","last_clean":"2024-03-16T12:19:44.547745+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:19:44.547745+0000","last_undegraded":"2024-03-16T12:19:44.547745+0000","last_fullsized":"2024-03-16T12:19:44.547745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:32.577303+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:20:32.577303+0000","last_peered":"2024-03-16T12:20:32.577303+0000","last_clean":"2024-03-16T12:20:32.577303+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:20:32.577303+0000","last_undegraded":"2024-03-16T12:20:32.577303+0000","last_fullsized":"2024-03-16T12:20:32.577303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774617+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:20:12.774617+0000","last_peered":"2024-03-16T12:20:12.774617+0000","last_clean":"2024-03-16T12:20:12.774617+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:20:12.774617+0000","last_undegraded":"2024-03-16T12:20:12.774617+0000","last_fullsized":"2024-03-16T12:20:12.774617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547967+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:19:44.547967+0000","last_peered":"2024-03-16T12:19:44.547967+0000","last_clean":"2024-03-16T12:19:44.547967+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:19:44.547967+0000","last_undegraded":"2024-03-16T12:19:44.547967+0000","last_fullsized":"2024-03-16T12:19:44.547967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547367+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:19:44.547367+0000","last_peered":"2024-03-16T12:19:44.547367+0000","last_clean":"2024-03-16T12:19:44.547367+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:19:44.547367+0000","last_undegraded":"2024-03-16T12:19:44.547367+0000","last_fullsized":"2024-03-16T12:19:44.547367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792543+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:20:12.792543+0000","last_peered":"2024-03-16T12:20:12.792543+0000","last_clean":"2024-03-16T12:20:12.792543+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:20:12.792543+0000","last_undegraded":"2024-03-16T12:20:12.792543+0000","last_fullsized":"2024-03-16T12:20:12.792543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545566+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:19:44.545566+0000","last_peered":"2024-03-16T12:19:44.545566+0000","last_clean":"2024-03-16T12:19:44.545566+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:19:44.545566+0000","last_undegraded":"2024-03-16T12:19:44.545566+0000","last_fullsized":"2024-03-16T12:19:44.545566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555845+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:44.555845+0000","last_peered":"2024-03-16T12:19:44.555845+0000","last_clean":"2024-03-16T12:19:44.555845+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:44.555845+0000","last_undegraded":"2024-03-16T12:19:44.555845+0000","last_fullsized":"2024-03-16T12:19:44.555845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792861+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:20:12.792861+0000","last_peered":"2024-03-16T12:20:12.792861+0000","last_clean":"2024-03-16T12:20:12.792861+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:20:12.792861+0000","last_undegraded":"2024-03-16T12:20:12.792861+0000","last_fullsized":"2024-03-16T12:20:12.792861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172253+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:19:45.172253+0000","last_peered":"2024-03-16T12:19:45.172253+0000","last_clean":"2024-03-16T12:19:45.172253+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:19:45.172253+0000","last_undegraded":"2024-03-16T12:19:45.172253+0000","last_fullsized":"2024-03-16T12:19:45.172253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549520+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:19:44.549520+0000","last_peered":"2024-03-16T12:19:44.549520+0000","last_clean":"2024-03-16T12:19:44.549520+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:19:44.549520+0000","last_undegraded":"2024-03-16T12:19:44.549520+0000","last_fullsized":"2024-03-16T12:19:44.549520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.801683+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:20:12.801683+0000","last_peered":"2024-03-16T12:20:12.801683+0000","last_clean":"2024-03-16T12:20:12.801683+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:20:12.801683+0000","last_undegraded":"2024-03-16T12:20:12.801683+0000","last_fullsized":"2024-03-16T12:20:12.801683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546386+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:19:44.546386+0000","last_peered":"2024-03-16T12:19:44.546386+0000","last_clean":"2024-03-16T12:19:44.546386+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:19:44.546386+0000","last_undegraded":"2024-03-16T12:19:44.546386+0000","last_fullsized":"2024-03-16T12:19:44.546386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:30.574776+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:20:30.574776+0000","last_peered":"2024-03-16T12:20:30.574776+0000","last_clean":"2024-03-16T12:20:30.574776+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:20:30.574776+0000","last_undegraded":"2024-03-16T12:20:30.574776+0000","last_fullsized":"2024-03-16T12:20:30.574776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787246+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:20:12.787246+0000","last_peered":"2024-03-16T12:20:12.787246+0000","last_clean":"2024-03-16T12:20:12.787246+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:20:12.787246+0000","last_undegraded":"2024-03-16T12:20:12.787246+0000","last_fullsized":"2024-03-16T12:20:12.787246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845520+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:19:44.845520+0000","last_peered":"2024-03-16T12:19:44.845520+0000","last_clean":"2024-03-16T12:19:44.845520+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:19:44.845520+0000","last_undegraded":"2024-03-16T12:19:44.845520+0000","last_fullsized":"2024-03-16T12:19:44.845520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548079+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:44.548079+0000","last_peered":"2024-03-16T12:19:44.548079+0000","last_clean":"2024-03-16T12:19:44.548079+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:44.548079+0000","last_undegraded":"2024-03-16T12:19:44.548079+0000","last_fullsized":"2024-03-16T12:19:44.548079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794077+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:20:12.794077+0000","last_peered":"2024-03-16T12:20:12.794077+0000","last_clean":"2024-03-16T12:20:12.794077+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:20:12.794077+0000","last_undegraded":"2024-03-16T12:20:12.794077+0000","last_fullsized":"2024-03-16T12:20:12.794077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546550+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:19:44.546550+0000","last_peered":"2024-03-16T12:19:44.546550+0000","last_clean":"2024-03-16T12:19:44.546550+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:19:44.546550+0000","last_undegraded":"2024-03-16T12:19:44.546550+0000","last_fullsized":"2024-03-16T12:19:44.546550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549429+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:44.549429+0000","last_peered":"2024-03-16T12:19:44.549429+0000","last_clean":"2024-03-16T12:19:44.549429+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:44.549429+0000","last_undegraded":"2024-03-16T12:19:44.549429+0000","last_fullsized":"2024-03-16T12:19:44.549429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785764+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:20:12.785764+0000","last_peered":"2024-03-16T12:20:12.785764+0000","last_clean":"2024-03-16T12:20:12.785764+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:20:12.785764+0000","last_undegraded":"2024-03-16T12:20:12.785764+0000","last_fullsized":"2024-03-16T12:20:12.785764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845603+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:19:44.845603+0000","last_peered":"2024-03-16T12:19:44.845603+0000","last_clean":"2024-03-16T12:19:44.845603+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:19:44.845603+0000","last_undegraded":"2024-03-16T12:19:44.845603+0000","last_fullsized":"2024-03-16T12:19:44.845603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":243,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:31.575771+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:20:31.575771+0000","last_peered":"2024-03-16T12:20:31.575771+0000","last_clean":"2024-03-16T12:20:31.575771+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:20:31.575771+0000","last_undegraded":"2024-03-16T12:20:31.575771+0000","last_fullsized":"2024-03-16T12:20:31.575771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":179,"num_read_kb":150,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.753013+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:20:12.753013+0000","last_peered":"2024-03-16T12:20:12.753013+0000","last_clean":"2024-03-16T12:20:12.753013+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:20:12.753013+0000","last_undegraded":"2024-03-16T12:20:12.753013+0000","last_fullsized":"2024-03-16T12:20:12.753013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550165+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:19:44.550165+0000","last_peered":"2024-03-16T12:19:44.550165+0000","last_clean":"2024-03-16T12:19:44.550165+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:19:44.550165+0000","last_undegraded":"2024-03-16T12:19:44.550165+0000","last_fullsized":"2024-03-16T12:19:44.550165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565318+0000","last_change":"2024-03-16T12:19:44.565318+0000","last_active":"2024-03-16T12:19:44.565318+0000","last_peered":"2024-03-16T12:19:44.565318+0000","last_clean":"2024-03-16T12:19:44.565318+0000","last_became_active":"2024-03-16T12:19:44.551024+0000","last_became_peered":"2024-03-16T12:19:44.551024+0000","last_unstale":"2024-03-16T12:19:44.565318+0000","last_undegraded":"2024-03-16T12:19:44.565318+0000","last_fullsized":"2024-03-16T12:19:44.565318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:15.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800239+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:20:12.800239+0000","last_peered":"2024-03-16T12:20:12.800239+0000","last_clean":"2024-03-16T12:20:12.800239+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:20:12.800239+0000","last_undegraded":"2024-03-16T12:20:12.800239+0000","last_fullsized":"2024-03-16T12:20:12.800239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.544965+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:19:44.544965+0000","last_peered":"2024-03-16T12:19:44.544965+0000","last_clean":"2024-03-16T12:19:44.544965+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:19:44.544965+0000","last_undegraded":"2024-03-16T12:19:44.544965+0000","last_fullsized":"2024-03-16T12:19:44.544965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549124+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:44.549124+0000","last_peered":"2024-03-16T12:19:44.549124+0000","last_clean":"2024-03-16T12:19:44.549124+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:44.549124+0000","last_undegraded":"2024-03-16T12:19:44.549124+0000","last_fullsized":"2024-03-16T12:19:44.549124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790226+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:20:12.790226+0000","last_peered":"2024-03-16T12:20:12.790226+0000","last_clean":"2024-03-16T12:20:12.790226+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:20:12.790226+0000","last_undegraded":"2024-03-16T12:20:12.790226+0000","last_fullsized":"2024-03-16T12:20:12.790226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.846915+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:19:44.846915+0000","last_peered":"2024-03-16T12:19:44.846915+0000","last_clean":"2024-03-16T12:19:44.846915+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:19:44.846915+0000","last_undegraded":"2024-03-16T12:19:44.846915+0000","last_fullsized":"2024-03-16T12:19:44.846915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:32.628969+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:20:32.628969+0000","last_peered":"2024-03-16T12:20:32.628969+0000","last_clean":"2024-03-16T12:20:32.628969+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:20:32.628969+0000","last_undegraded":"2024-03-16T12:20:32.628969+0000","last_fullsized":"2024-03-16T12:20:32.628969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.754234+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:20:12.754234+0000","last_peered":"2024-03-16T12:20:12.754234+0000","last_clean":"2024-03-16T12:20:12.754234+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:20:12.754234+0000","last_undegraded":"2024-03-16T12:20:12.754234+0000","last_fullsized":"2024-03-16T12:20:12.754234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565998+0000","last_change":"2024-03-16T12:19:44.565998+0000","last_active":"2024-03-16T12:19:44.565998+0000","last_peered":"2024-03-16T12:19:44.565998+0000","last_clean":"2024-03-16T12:19:44.565998+0000","last_became_active":"2024-03-16T12:19:44.564591+0000","last_became_peered":"2024-03-16T12:19:44.564591+0000","last_unstale":"2024-03-16T12:19:44.565998+0000","last_undegraded":"2024-03-16T12:19:44.565998+0000","last_fullsized":"2024-03-16T12:19:44.565998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:57:48.227439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543302+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:44.543302+0000","last_peered":"2024-03-16T12:19:44.543302+0000","last_clean":"2024-03-16T12:19:44.543302+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:44.543302+0000","last_undegraded":"2024-03-16T12:19:44.543302+0000","last_fullsized":"2024-03-16T12:19:44.543302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782528+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:20:12.782528+0000","last_peered":"2024-03-16T12:20:12.782528+0000","last_clean":"2024-03-16T12:20:12.782528+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:20:12.782528+0000","last_undegraded":"2024-03-16T12:20:12.782528+0000","last_fullsized":"2024-03-16T12:20:12.782528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548883+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:19:44.548883+0000","last_peered":"2024-03-16T12:19:44.548883+0000","last_clean":"2024-03-16T12:19:44.548883+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:19:44.548883+0000","last_undegraded":"2024-03-16T12:19:44.548883+0000","last_fullsized":"2024-03-16T12:19:44.548883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543827+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:44.543827+0000","last_peered":"2024-03-16T12:19:44.543827+0000","last_clean":"2024-03-16T12:19:44.543827+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:44.543827+0000","last_undegraded":"2024-03-16T12:19:44.543827+0000","last_fullsized":"2024-03-16T12:19:44.543827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.768313+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:20:12.768313+0000","last_peered":"2024-03-16T12:20:12.768313+0000","last_clean":"2024-03-16T12:20:12.768313+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:20:12.768313+0000","last_undegraded":"2024-03-16T12:20:12.768313+0000","last_fullsized":"2024-03-16T12:20:12.768313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543432+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:19:44.543432+0000","last_peered":"2024-03-16T12:19:44.543432+0000","last_clean":"2024-03-16T12:19:44.543432+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:19:44.543432+0000","last_undegraded":"2024-03-16T12:19:44.543432+0000","last_fullsized":"2024-03-16T12:19:44.543432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847386+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:19:44.847386+0000","last_peered":"2024-03-16T12:19:44.847386+0000","last_clean":"2024-03-16T12:19:44.847386+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:19:44.847386+0000","last_undegraded":"2024-03-16T12:19:44.847386+0000","last_fullsized":"2024-03-16T12:19:44.847386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793024+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:20:12.793024+0000","last_peered":"2024-03-16T12:20:12.793024+0000","last_clean":"2024-03-16T12:20:12.793024+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:20:12.793024+0000","last_undegraded":"2024-03-16T12:20:12.793024+0000","last_fullsized":"2024-03-16T12:20:12.793024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.557767+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:19:44.557767+0000","last_peered":"2024-03-16T12:19:44.557767+0000","last_clean":"2024-03-16T12:19:44.557767+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:19:44.557767+0000","last_undegraded":"2024-03-16T12:19:44.557767+0000","last_fullsized":"2024-03-16T12:19:44.557767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547765+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:19:44.547765+0000","last_peered":"2024-03-16T12:19:44.547765+0000","last_clean":"2024-03-16T12:19:44.547765+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:19:44.547765+0000","last_undegraded":"2024-03-16T12:19:44.547765+0000","last_fullsized":"2024-03-16T12:19:44.547765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786168+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:20:12.786168+0000","last_peered":"2024-03-16T12:20:12.786168+0000","last_clean":"2024-03-16T12:20:12.786168+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:20:12.786168+0000","last_undegraded":"2024-03-16T12:20:12.786168+0000","last_fullsized":"2024-03-16T12:20:12.786168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559339+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:19:44.559339+0000","last_peered":"2024-03-16T12:19:44.559339+0000","last_clean":"2024-03-16T12:19:44.559339+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:19:44.559339+0000","last_undegraded":"2024-03-16T12:19:44.559339+0000","last_fullsized":"2024-03-16T12:19:44.559339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776386+0000","last_change":"2024-03-16T12:19:44.552241+0000","last_active":"2024-03-16T12:20:12.776386+0000","last_peered":"2024-03-16T12:20:12.776386+0000","last_clean":"2024-03-16T12:20:12.776386+0000","last_became_active":"2024-03-16T12:19:44.551762+0000","last_became_peered":"2024-03-16T12:19:44.551762+0000","last_unstale":"2024-03-16T12:20:12.776386+0000","last_undegraded":"2024-03-16T12:20:12.776386+0000","last_fullsized":"2024-03-16T12:20:12.776386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:24.454587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:32.628398+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:20:32.628398+0000","last_peered":"2024-03-16T12:20:32.628398+0000","last_clean":"2024-03-16T12:20:32.628398+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:20:32.628398+0000","last_undegraded":"2024-03-16T12:20:32.628398+0000","last_fullsized":"2024-03-16T12:20:32.628398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547939+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:19:44.547939+0000","last_peered":"2024-03-16T12:19:44.547939+0000","last_clean":"2024-03-16T12:19:44.547939+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:19:44.547939+0000","last_undegraded":"2024-03-16T12:19:44.547939+0000","last_fullsized":"2024-03-16T12:19:44.547939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790228+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:20:12.790228+0000","last_peered":"2024-03-16T12:20:12.790228+0000","last_clean":"2024-03-16T12:20:12.790228+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:20:12.790228+0000","last_undegraded":"2024-03-16T12:20:12.790228+0000","last_fullsized":"2024-03-16T12:20:12.790228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546475+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:44.546475+0000","last_peered":"2024-03-16T12:19:44.546475+0000","last_clean":"2024-03-16T12:19:44.546475+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:44.546475+0000","last_undegraded":"2024-03-16T12:19:44.546475+0000","last_fullsized":"2024-03-16T12:19:44.546475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548608+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:19:44.548608+0000","last_peered":"2024-03-16T12:19:44.548608+0000","last_clean":"2024-03-16T12:19:44.548608+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:19:44.548608+0000","last_undegraded":"2024-03-16T12:19:44.548608+0000","last_fullsized":"2024-03-16T12:19:44.548608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.566987+0000","last_change":"2024-03-16T12:19:44.566987+0000","last_active":"2024-03-16T12:19:44.566987+0000","last_peered":"2024-03-16T12:19:44.566987+0000","last_clean":"2024-03-16T12:19:44.566987+0000","last_became_active":"2024-03-16T12:19:44.566529+0000","last_became_peered":"2024-03-16T12:19:44.566529+0000","last_unstale":"2024-03-16T12:19:44.566987+0000","last_undegraded":"2024-03-16T12:19:44.566987+0000","last_fullsized":"2024-03-16T12:19:44.566987+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:53:03.124434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785242+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:20:12.785242+0000","last_peered":"2024-03-16T12:20:12.785242+0000","last_clean":"2024-03-16T12:20:12.785242+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:20:12.785242+0000","last_undegraded":"2024-03-16T12:20:12.785242+0000","last_fullsized":"2024-03-16T12:20:12.785242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548660+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:44.548660+0000","last_peered":"2024-03-16T12:19:44.548660+0000","last_clean":"2024-03-16T12:19:44.548660+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:44.548660+0000","last_undegraded":"2024-03-16T12:19:44.548660+0000","last_fullsized":"2024-03-16T12:19:44.548660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559544+0000","last_change":"2024-03-16T12:19:44.559544+0000","last_active":"2024-03-16T12:19:44.559544+0000","last_peered":"2024-03-16T12:19:44.559544+0000","last_clean":"2024-03-16T12:19:44.559544+0000","last_became_active":"2024-03-16T12:19:44.559030+0000","last_became_peered":"2024-03-16T12:19:44.559030+0000","last_unstale":"2024-03-16T12:19:44.559544+0000","last_undegraded":"2024-03-16T12:19:44.559544+0000","last_fullsized":"2024-03-16T12:19:44.559544+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:46.496512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7,6],"acting":[11,10,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543688+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:44.543688+0000","last_peered":"2024-03-16T12:19:44.543688+0000","last_clean":"2024-03-16T12:19:44.543688+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:44.543688+0000","last_undegraded":"2024-03-16T12:19:44.543688+0000","last_fullsized":"2024-03-16T12:19:44.543688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786161+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:20:12.786161+0000","last_peered":"2024-03-16T12:20:12.786161+0000","last_clean":"2024-03-16T12:20:12.786161+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:20:12.786161+0000","last_undegraded":"2024-03-16T12:20:12.786161+0000","last_fullsized":"2024-03-16T12:20:12.786161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546367+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:19:44.546367+0000","last_peered":"2024-03-16T12:19:44.546367+0000","last_clean":"2024-03-16T12:19:44.546367+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:19:44.546367+0000","last_undegraded":"2024-03-16T12:19:44.546367+0000","last_fullsized":"2024-03-16T12:19:44.546367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847757+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:19:44.847757+0000","last_peered":"2024-03-16T12:19:44.847757+0000","last_clean":"2024-03-16T12:19:44.847757+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:19:44.847757+0000","last_undegraded":"2024-03-16T12:19:44.847757+0000","last_fullsized":"2024-03-16T12:19:44.847757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.783807+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:20:12.783807+0000","last_peered":"2024-03-16T12:20:12.783807+0000","last_clean":"2024-03-16T12:20:12.783807+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:20:12.783807+0000","last_undegraded":"2024-03-16T12:20:12.783807+0000","last_fullsized":"2024-03-16T12:20:12.783807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546775+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:19:44.546775+0000","last_peered":"2024-03-16T12:19:44.546775+0000","last_clean":"2024-03-16T12:19:44.546775+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:19:44.546775+0000","last_undegraded":"2024-03-16T12:19:44.546775+0000","last_fullsized":"2024-03-16T12:19:44.546775+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546916+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:44.546916+0000","last_peered":"2024-03-16T12:19:44.546916+0000","last_clean":"2024-03-16T12:19:44.546916+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:44.546916+0000","last_undegraded":"2024-03-16T12:19:44.546916+0000","last_fullsized":"2024-03-16T12:19:44.546916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804835+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:20:12.804835+0000","last_peered":"2024-03-16T12:20:12.804835+0000","last_clean":"2024-03-16T12:20:12.804835+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:20:12.804835+0000","last_undegraded":"2024-03-16T12:20:12.804835+0000","last_fullsized":"2024-03-16T12:20:12.804835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546439+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:19:44.546439+0000","last_peered":"2024-03-16T12:19:44.546439+0000","last_clean":"2024-03-16T12:19:44.546439+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:19:44.546439+0000","last_undegraded":"2024-03-16T12:19:44.546439+0000","last_fullsized":"2024-03-16T12:19:44.546439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168814+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:45.168814+0000","last_peered":"2024-03-16T12:19:45.168814+0000","last_clean":"2024-03-16T12:19:45.168814+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:45.168814+0000","last_undegraded":"2024-03-16T12:19:45.168814+0000","last_fullsized":"2024-03-16T12:19:45.168814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.740265+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:20:12.740265+0000","last_peered":"2024-03-16T12:20:12.740265+0000","last_clean":"2024-03-16T12:20:12.740265+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:20:12.740265+0000","last_undegraded":"2024-03-16T12:20:12.740265+0000","last_fullsized":"2024-03-16T12:20:12.740265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847128+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:19:44.847128+0000","last_peered":"2024-03-16T12:19:44.847128+0000","last_clean":"2024-03-16T12:19:44.847128+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:19:44.847128+0000","last_undegraded":"2024-03-16T12:19:44.847128+0000","last_fullsized":"2024-03-16T12:19:44.847128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546423+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:44.546423+0000","last_peered":"2024-03-16T12:19:44.546423+0000","last_clean":"2024-03-16T12:19:44.546423+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:44.546423+0000","last_undegraded":"2024-03-16T12:19:44.546423+0000","last_fullsized":"2024-03-16T12:19:44.546423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803410+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:20:12.803410+0000","last_peered":"2024-03-16T12:20:12.803410+0000","last_clean":"2024-03-16T12:20:12.803410+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:20:12.803410+0000","last_undegraded":"2024-03-16T12:20:12.803410+0000","last_fullsized":"2024-03-16T12:20:12.803410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546286+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:19:44.546286+0000","last_peered":"2024-03-16T12:19:44.546286+0000","last_clean":"2024-03-16T12:19:44.546286+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:19:44.546286+0000","last_undegraded":"2024-03-16T12:19:44.546286+0000","last_fullsized":"2024-03-16T12:19:44.546286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547011+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:19:44.547011+0000","last_peered":"2024-03-16T12:19:44.547011+0000","last_clean":"2024-03-16T12:19:44.547011+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:19:44.547011+0000","last_undegraded":"2024-03-16T12:19:44.547011+0000","last_fullsized":"2024-03-16T12:19:44.547011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804622+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:20:12.804622+0000","last_peered":"2024-03-16T12:20:12.804622+0000","last_clean":"2024-03-16T12:20:12.804622+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:20:12.804622+0000","last_undegraded":"2024-03-16T12:20:12.804622+0000","last_fullsized":"2024-03-16T12:20:12.804622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546286+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:19:44.546286+0000","last_peered":"2024-03-16T12:19:44.546286+0000","last_clean":"2024-03-16T12:19:44.546286+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:19:44.546286+0000","last_undegraded":"2024-03-16T12:19:44.546286+0000","last_fullsized":"2024-03-16T12:19:44.546286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555974+0000","last_change":"2024-03-16T12:19:44.555974+0000","last_active":"2024-03-16T12:19:44.555974+0000","last_peered":"2024-03-16T12:19:44.555974+0000","last_clean":"2024-03-16T12:19:44.555974+0000","last_became_active":"2024-03-16T12:19:44.550254+0000","last_became_peered":"2024-03-16T12:19:44.550254+0000","last_unstale":"2024-03-16T12:19:44.555974+0000","last_undegraded":"2024-03-16T12:19:44.555974+0000","last_fullsized":"2024-03-16T12:19:44.555974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789419+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:20:12.789419+0000","last_peered":"2024-03-16T12:20:12.789419+0000","last_clean":"2024-03-16T12:20:12.789419+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:20:12.789419+0000","last_undegraded":"2024-03-16T12:20:12.789419+0000","last_fullsized":"2024-03-16T12:20:12.789419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168826+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:19:45.168826+0000","last_peered":"2024-03-16T12:19:45.168826+0000","last_clean":"2024-03-16T12:19:45.168826+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:19:45.168826+0000","last_undegraded":"2024-03-16T12:19:45.168826+0000","last_fullsized":"2024-03-16T12:19:45.168826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548123+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:19:44.548123+0000","last_peered":"2024-03-16T12:19:44.548123+0000","last_clean":"2024-03-16T12:19:44.548123+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:19:44.548123+0000","last_undegraded":"2024-03-16T12:19:44.548123+0000","last_fullsized":"2024-03-16T12:19:44.548123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786614+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:20:12.786614+0000","last_peered":"2024-03-16T12:20:12.786614+0000","last_clean":"2024-03-16T12:20:12.786614+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:20:12.786614+0000","last_undegraded":"2024-03-16T12:20:12.786614+0000","last_fullsized":"2024-03-16T12:20:12.786614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.171930+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:19:45.171930+0000","last_peered":"2024-03-16T12:19:45.171930+0000","last_clean":"2024-03-16T12:19:45.171930+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:19:45.171930+0000","last_undegraded":"2024-03-16T12:19:45.171930+0000","last_fullsized":"2024-03-16T12:19:45.171930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549942+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:19:44.549942+0000","last_peered":"2024-03-16T12:19:44.549942+0000","last_clean":"2024-03-16T12:19:44.549942+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:19:44.549942+0000","last_undegraded":"2024-03-16T12:19:44.549942+0000","last_fullsized":"2024-03-16T12:19:44.549942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788679+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:20:12.788679+0000","last_peered":"2024-03-16T12:20:12.788679+0000","last_clean":"2024-03-16T12:20:12.788679+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:20:12.788679+0000","last_undegraded":"2024-03-16T12:20:12.788679+0000","last_fullsized":"2024-03-16T12:20:12.788679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549223+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:19:44.549223+0000","last_peered":"2024-03-16T12:19:44.549223+0000","last_clean":"2024-03-16T12:19:44.549223+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:19:44.549223+0000","last_undegraded":"2024-03-16T12:19:44.549223+0000","last_fullsized":"2024-03-16T12:19:44.549223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559443+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:19:44.559443+0000","last_peered":"2024-03-16T12:19:44.559443+0000","last_clean":"2024-03-16T12:19:44.559443+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:19:44.559443+0000","last_undegraded":"2024-03-16T12:19:44.559443+0000","last_fullsized":"2024-03-16T12:19:44.559443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.771593+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:20:12.771593+0000","last_peered":"2024-03-16T12:20:12.771593+0000","last_clean":"2024-03-16T12:20:12.771593+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:20:12.771593+0000","last_undegraded":"2024-03-16T12:20:12.771593+0000","last_fullsized":"2024-03-16T12:20:12.771593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774110+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:20:12.774110+0000","last_peered":"2024-03-16T12:20:12.774110+0000","last_clean":"2024-03-16T12:20:12.774110+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:20:12.774110+0000","last_undegraded":"2024-03-16T12:20:12.774110+0000","last_fullsized":"2024-03-16T12:20:12.774110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.780975+0000","last_change":"2024-03-16T12:19:44.847899+0000","last_active":"2024-03-16T12:20:12.780975+0000","last_peered":"2024-03-16T12:20:12.780975+0000","last_clean":"2024-03-16T12:20:12.780975+0000","last_became_active":"2024-03-16T12:19:44.847589+0000","last_became_peered":"2024-03-16T12:19:44.847589+0000","last_unstale":"2024-03-16T12:20:12.780975+0000","last_undegraded":"2024-03-16T12:20:12.780975+0000","last_fullsized":"2024-03-16T12:20:12.780975+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:59.091516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.802745+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:20:12.802745+0000","last_peered":"2024-03-16T12:20:12.802745+0000","last_clean":"2024-03-16T12:20:12.802745+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:20:12.802745+0000","last_undegraded":"2024-03-16T12:20:12.802745+0000","last_fullsized":"2024-03-16T12:20:12.802745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804406+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:20:12.804406+0000","last_peered":"2024-03-16T12:20:12.804406+0000","last_clean":"2024-03-16T12:20:12.804406+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:20:12.804406+0000","last_undegraded":"2024-03-16T12:20:12.804406+0000","last_fullsized":"2024-03-16T12:20:12.804406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804088+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:20:12.804088+0000","last_peered":"2024-03-16T12:20:12.804088+0000","last_clean":"2024-03-16T12:20:12.804088+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:20:12.804088+0000","last_undegraded":"2024-03-16T12:20:12.804088+0000","last_fullsized":"2024-03-16T12:20:12.804088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770383+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:20:12.770383+0000","last_peered":"2024-03-16T12:20:12.770383+0000","last_clean":"2024-03-16T12:20:12.770383+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:20:12.770383+0000","last_undegraded":"2024-03-16T12:20:12.770383+0000","last_fullsized":"2024-03-16T12:20:12.770383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776817+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:20:12.776817+0000","last_peered":"2024-03-16T12:20:12.776817+0000","last_clean":"2024-03-16T12:20:12.776817+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:20:12.776817+0000","last_undegraded":"2024-03-16T12:20:12.776817+0000","last_fullsized":"2024-03-16T12:20:12.776817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776082+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:20:12.776082+0000","last_peered":"2024-03-16T12:20:12.776082+0000","last_clean":"2024-03-16T12:20:12.776082+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:20:12.776082+0000","last_undegraded":"2024-03-16T12:20:12.776082+0000","last_fullsized":"2024-03-16T12:20:12.776082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799884+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:20:12.799884+0000","last_peered":"2024-03-16T12:20:12.799884+0000","last_clean":"2024-03-16T12:20:12.799884+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:20:12.799884+0000","last_undegraded":"2024-03-16T12:20:12.799884+0000","last_fullsized":"2024-03-16T12:20:12.799884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.760626+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:20:12.760626+0000","last_peered":"2024-03-16T12:20:12.760626+0000","last_clean":"2024-03-16T12:20:12.760626+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:20:12.760626+0000","last_undegraded":"2024-03-16T12:20:12.760626+0000","last_fullsized":"2024-03-16T12:20:12.760626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800202+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:20:12.800202+0000","last_peered":"2024-03-16T12:20:12.800202+0000","last_clean":"2024-03-16T12:20:12.800202+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:20:12.800202+0000","last_undegraded":"2024-03-16T12:20:12.800202+0000","last_fullsized":"2024-03-16T12:20:12.800202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.806499+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:20:12.806499+0000","last_peered":"2024-03-16T12:20:12.806499+0000","last_clean":"2024-03-16T12:20:12.806499+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:20:12.806499+0000","last_undegraded":"2024-03-16T12:20:12.806499+0000","last_fullsized":"2024-03-16T12:20:12.806499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781457+0000","last_change":"2024-03-16T12:19:44.849288+0000","last_active":"2024-03-16T12:20:12.781457+0000","last_peered":"2024-03-16T12:20:12.781457+0000","last_clean":"2024-03-16T12:20:12.781457+0000","last_became_active":"2024-03-16T12:19:44.848869+0000","last_became_peered":"2024-03-16T12:19:44.848869+0000","last_unstale":"2024-03-16T12:20:12.781457+0000","last_undegraded":"2024-03-16T12:20:12.781457+0000","last_fullsized":"2024-03-16T12:20:12.781457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:36.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66192,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":541,"num_read_kb":338,"num_write":169,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":198576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":500849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28692,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716460,"statfs":{"total":95995035648,"available":95965655040,"internally_reserved":0,"allocated":1716224,"data_stored":1150665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":500971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1096489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":500877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1092077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:20:36.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T12:20:37.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:36 smithi098 ceph-mon[31633]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:37.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:36 smithi153 ceph-mon[36901]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:37.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:36 smithi173 ceph-mon[36632]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:37.357 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:38.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:37 smithi098 ceph-mon[31633]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:38.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:37 smithi153 ceph-mon[36901]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:38.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:37 smithi173 ceph-mon[36632]: pgmap v378: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 -- 172.21.15.98:0/3000784030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e8104e80 msgr2=0x7fd1e8111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3000784030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e8104e80 0x7fd1e8111340 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000b0d0 tx=0x7fd1e000b3e0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 -- 172.21.15.98:0/3000784030 shutdown_connections 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3000784030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e8104e80 0x7fd1e8111340 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3000784030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e810a570 0x7fd1e810a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3000784030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 0x7fd1e8104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 -- 172.21.15.98:0/3000784030 >> 172.21.15.98:0/3000784030 conn(0x7fd1e80fffb0 msgr2=0x7fd1e81023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 -- 172.21.15.98:0/3000784030 shutdown_connections 2024-03-16T12:20:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.631+0000 7fd1edc00700 1 -- 172.21.15.98:0/3000784030 wait complete. 2024-03-16T12:20:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.632+0000 7fd1edc00700 1 Processor -- start 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 -- start start 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 0x7fd1e81a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e81147a0 con 0x7fd1e81044c0 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e8114620 con 0x7fd1e810a570 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1edc00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e8114920 con 0x7fd1e8104e80 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e6a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 0x7fd1e81a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e6000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e6000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57544/0 (socket says 172.21.15.98:57544) 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e7400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33290/0 (socket says 172.21.15.98:33290) 2024-03-16T12:20:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.633+0000 7fd1e6000700 1 -- 172.21.15.98:0/3348022122 learned_addr learned my addr 172.21.15.98:0/3348022122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 -- 172.21.15.98:0/3348022122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 msgr2=0x7fd1e81a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 -- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 msgr2=0x7fd1e81a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 0x7fd1e81a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 -- 172.21.15.98:0/3348022122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e819b0e0 con 0x7fd1e810a570 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e6000700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1e7400700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000b770 tx=0x7fd1e0007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0021020 con 0x7fd1e810a570 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1e0004c50 con 0x7fd1e810a570 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.634+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00052b0 con 0x7fd1e810a570 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.635+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e819b370 con 0x7fd1e810a570 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.635+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e81a7640 con 0x7fd1e810a570 2024-03-16T12:20:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.636+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e8105be0 con 0x7fd1e810a570 2024-03-16T12:20:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.638+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd1e0024020 con 0x7fd1e810a570 2024-03-16T12:20:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.639+0000 7fd1df400700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 0x7fd1cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.639+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1e000f020 con 0x7fd1e810a570 2024-03-16T12:20:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.639+0000 7fd1e6a00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 0x7fd1cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.640+0000 7fd1e6a00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 0x7fd1cc0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1e81094e0 tx=0x7fd1d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:39.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.647+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd1e009c9e0 con 0x7fd1e810a570 2024-03-16T12:20:39.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.863+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1e8062710 con 0x7fd1cc0a2910 2024-03-16T12:20:39.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.893+0000 7fd1df400700 1 -- 172.21.15.98:0/3348022122 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340483 (secure 0 0 0) 0x7fd1e8062710 con 0x7fd1cc0a2910 2024-03-16T12:20:39.881 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:39.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 msgr2=0x7fd1cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:39.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 0x7fd1cc0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1e81094e0 tx=0x7fd1d8009260 comp rx=0 tx=0).stop 2024-03-16T12:20:39.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 msgr2=0x7fd1e819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000b770 tx=0x7fd1e0007a20 comp rx=0 tx=0).stop 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 shutdown_connections 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd1cc0a2910 0x7fd1cc0a4dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1e810a570 0x7fd1e819a980 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e81044c0 0x7fd1e81a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 --2- 172.21.15.98:0/3348022122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1e8104e80 0x7fd1e81a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 >> 172.21.15.98:0/3348022122 conn(0x7fd1e80fffb0 msgr2=0x7fd1e8101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 shutdown_connections 2024-03-16T12:20:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:39.902+0000 7fd1edc00700 1 -- 172.21.15.98:0/3348022122 wait complete. 2024-03-16T12:20:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:20:40.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:39 smithi098 ceph-mon[31633]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:40.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:39 smithi153 ceph-mon[36901]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:40.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:39 smithi173 ceph-mon[36632]: pgmap v379: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:40.705 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-03-16T12:20:38.824884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656794,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":687,"num_read_kb":457,"num_write":352,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384048,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557776,"statfs":{"total":1151940427776,"available":1151547162624,"internally_reserved":0,"allocated":14434304,"data_stored":7956092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005281"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803695+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:20:12.803695+0000","last_peered":"2024-03-16T12:20:12.803695+0000","last_clean":"2024-03-16T12:20:12.803695+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:20:12.803695+0000","last_undegraded":"2024-03-16T12:20:12.803695+0000","last_fullsized":"2024-03-16T12:20:12.803695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798064+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:20:12.798064+0000","last_peered":"2024-03-16T12:20:12.798064+0000","last_clean":"2024-03-16T12:20:12.798064+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:20:12.798064+0000","last_undegraded":"2024-03-16T12:20:12.798064+0000","last_fullsized":"2024-03-16T12:20:12.798064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799952+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:20:12.799952+0000","last_peered":"2024-03-16T12:20:12.799952+0000","last_clean":"2024-03-16T12:20:12.799952+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:20:12.799952+0000","last_undegraded":"2024-03-16T12:20:12.799952+0000","last_fullsized":"2024-03-16T12:20:12.799952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755621+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:20:12.755621+0000","last_peered":"2024-03-16T12:20:12.755621+0000","last_clean":"2024-03-16T12:20:12.755621+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:20:12.755621+0000","last_undegraded":"2024-03-16T12:20:12.755621+0000","last_fullsized":"2024-03-16T12:20:12.755621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.778564+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:20:12.778564+0000","last_peered":"2024-03-16T12:20:12.778564+0000","last_clean":"2024-03-16T12:20:12.778564+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:20:12.778564+0000","last_undegraded":"2024-03-16T12:20:12.778564+0000","last_fullsized":"2024-03-16T12:20:12.778564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785812+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:20:12.785812+0000","last_peered":"2024-03-16T12:20:12.785812+0000","last_clean":"2024-03-16T12:20:12.785812+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:20:12.785812+0000","last_undegraded":"2024-03-16T12:20:12.785812+0000","last_fullsized":"2024-03-16T12:20:12.785812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.766436+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:20:12.766436+0000","last_peered":"2024-03-16T12:20:12.766436+0000","last_clean":"2024-03-16T12:20:12.766436+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:20:12.766436+0000","last_undegraded":"2024-03-16T12:20:12.766436+0000","last_fullsized":"2024-03-16T12:20:12.766436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793048+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:20:12.793048+0000","last_peered":"2024-03-16T12:20:12.793048+0000","last_clean":"2024-03-16T12:20:12.793048+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:20:12.793048+0000","last_undegraded":"2024-03-16T12:20:12.793048+0000","last_fullsized":"2024-03-16T12:20:12.793048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782148+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:20:12.782148+0000","last_peered":"2024-03-16T12:20:12.782148+0000","last_clean":"2024-03-16T12:20:12.782148+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:20:12.782148+0000","last_undegraded":"2024-03-16T12:20:12.782148+0000","last_fullsized":"2024-03-16T12:20:12.782148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785267+0000","last_change":"2024-03-16T12:19:44.848697+0000","last_active":"2024-03-16T12:20:12.785267+0000","last_peered":"2024-03-16T12:20:12.785267+0000","last_clean":"2024-03-16T12:20:12.785267+0000","last_became_active":"2024-03-16T12:19:44.848414+0000","last_became_peered":"2024-03-16T12:19:44.848414+0000","last_unstale":"2024-03-16T12:20:12.785267+0000","last_undegraded":"2024-03-16T12:20:12.785267+0000","last_fullsized":"2024-03-16T12:20:12.785267+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:54.721056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786301+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:20:12.786301+0000","last_peered":"2024-03-16T12:20:12.786301+0000","last_clean":"2024-03-16T12:20:12.786301+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:20:12.786301+0000","last_undegraded":"2024-03-16T12:20:12.786301+0000","last_fullsized":"2024-03-16T12:20:12.786301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.754844+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:20:12.754844+0000","last_peered":"2024-03-16T12:20:12.754844+0000","last_clean":"2024-03-16T12:20:12.754844+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:20:12.754844+0000","last_undegraded":"2024-03-16T12:20:12.754844+0000","last_fullsized":"2024-03-16T12:20:12.754844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790030+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:20:12.790030+0000","last_peered":"2024-03-16T12:20:12.790030+0000","last_clean":"2024-03-16T12:20:12.790030+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:20:12.790030+0000","last_undegraded":"2024-03-16T12:20:12.790030+0000","last_fullsized":"2024-03-16T12:20:12.790030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793934+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:20:12.793934+0000","last_peered":"2024-03-16T12:20:12.793934+0000","last_clean":"2024-03-16T12:20:12.793934+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:20:12.793934+0000","last_undegraded":"2024-03-16T12:20:12.793934+0000","last_fullsized":"2024-03-16T12:20:12.793934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.795833+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:20:12.795833+0000","last_peered":"2024-03-16T12:20:12.795833+0000","last_clean":"2024-03-16T12:20:12.795833+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:20:12.795833+0000","last_undegraded":"2024-03-16T12:20:12.795833+0000","last_fullsized":"2024-03-16T12:20:12.795833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803822+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:20:12.803822+0000","last_peered":"2024-03-16T12:20:12.803822+0000","last_clean":"2024-03-16T12:20:12.803822+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:20:12.803822+0000","last_undegraded":"2024-03-16T12:20:12.803822+0000","last_fullsized":"2024-03-16T12:20:12.803822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803923+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:20:12.803923+0000","last_peered":"2024-03-16T12:20:12.803923+0000","last_clean":"2024-03-16T12:20:12.803923+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:20:12.803923+0000","last_undegraded":"2024-03-16T12:20:12.803923+0000","last_fullsized":"2024-03-16T12:20:12.803923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786099+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:20:12.786099+0000","last_peered":"2024-03-16T12:20:12.786099+0000","last_clean":"2024-03-16T12:20:12.786099+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:20:12.786099+0000","last_undegraded":"2024-03-16T12:20:12.786099+0000","last_fullsized":"2024-03-16T12:20:12.786099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792800+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:20:12.792800+0000","last_peered":"2024-03-16T12:20:12.792800+0000","last_clean":"2024-03-16T12:20:12.792800+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:20:12.792800+0000","last_undegraded":"2024-03-16T12:20:12.792800+0000","last_fullsized":"2024-03-16T12:20:12.792800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547227+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:19:44.547227+0000","last_peered":"2024-03-16T12:19:44.547227+0000","last_clean":"2024-03-16T12:19:44.547227+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:19:44.547227+0000","last_undegraded":"2024-03-16T12:19:44.547227+0000","last_fullsized":"2024-03-16T12:19:44.547227+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847242+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:44.847242+0000","last_peered":"2024-03-16T12:19:44.847242+0000","last_clean":"2024-03-16T12:19:44.847242+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:44.847242+0000","last_undegraded":"2024-03-16T12:19:44.847242+0000","last_fullsized":"2024-03-16T12:19:44.847242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.767386+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:20:12.767386+0000","last_peered":"2024-03-16T12:20:12.767386+0000","last_clean":"2024-03-16T12:20:12.767386+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:20:12.767386+0000","last_undegraded":"2024-03-16T12:20:12.767386+0000","last_fullsized":"2024-03-16T12:20:12.767386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559192+0000","last_change":"2024-03-16T12:19:44.559192+0000","last_active":"2024-03-16T12:19:44.559192+0000","last_peered":"2024-03-16T12:19:44.559192+0000","last_clean":"2024-03-16T12:19:44.559192+0000","last_became_active":"2024-03-16T12:19:44.558795+0000","last_became_peered":"2024-03-16T12:19:44.558795+0000","last_unstale":"2024-03-16T12:19:44.559192+0000","last_undegraded":"2024-03-16T12:19:44.559192+0000","last_fullsized":"2024-03-16T12:19:44.559192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:41:28.267737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550779+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:19:44.550779+0000","last_peered":"2024-03-16T12:19:44.550779+0000","last_clean":"2024-03-16T12:19:44.550779+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:19:44.550779+0000","last_undegraded":"2024-03-16T12:19:44.550779+0000","last_fullsized":"2024-03-16T12:19:44.550779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803963+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:20:12.803963+0000","last_peered":"2024-03-16T12:20:12.803963+0000","last_clean":"2024-03-16T12:20:12.803963+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:20:12.803963+0000","last_undegraded":"2024-03-16T12:20:12.803963+0000","last_fullsized":"2024-03-16T12:20:12.803963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546335+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:19:44.546335+0000","last_peered":"2024-03-16T12:19:44.546335+0000","last_clean":"2024-03-16T12:19:44.546335+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:19:44.546335+0000","last_undegraded":"2024-03-16T12:19:44.546335+0000","last_fullsized":"2024-03-16T12:19:44.546335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543804+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:44.543804+0000","last_peered":"2024-03-16T12:19:44.543804+0000","last_clean":"2024-03-16T12:19:44.543804+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:44.543804+0000","last_undegraded":"2024-03-16T12:19:44.543804+0000","last_fullsized":"2024-03-16T12:19:44.543804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788655+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:20:12.788655+0000","last_peered":"2024-03-16T12:20:12.788655+0000","last_clean":"2024-03-16T12:20:12.788655+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:20:12.788655+0000","last_undegraded":"2024-03-16T12:20:12.788655+0000","last_fullsized":"2024-03-16T12:20:12.788655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545946+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:19:44.545946+0000","last_peered":"2024-03-16T12:19:44.545946+0000","last_clean":"2024-03-16T12:19:44.545946+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:19:44.545946+0000","last_undegraded":"2024-03-16T12:19:44.545946+0000","last_fullsized":"2024-03-16T12:19:44.545946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543884+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:19:44.543884+0000","last_peered":"2024-03-16T12:19:44.543884+0000","last_clean":"2024-03-16T12:19:44.543884+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:19:44.543884+0000","last_undegraded":"2024-03-16T12:19:44.543884+0000","last_fullsized":"2024-03-16T12:19:44.543884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.784747+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:20:12.784747+0000","last_peered":"2024-03-16T12:20:12.784747+0000","last_clean":"2024-03-16T12:20:12.784747+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:20:12.784747+0000","last_undegraded":"2024-03-16T12:20:12.784747+0000","last_fullsized":"2024-03-16T12:20:12.784747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547745+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:19:44.547745+0000","last_peered":"2024-03-16T12:19:44.547745+0000","last_clean":"2024-03-16T12:19:44.547745+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:19:44.547745+0000","last_undegraded":"2024-03-16T12:19:44.547745+0000","last_fullsized":"2024-03-16T12:19:44.547745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:37.582450+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:20:37.582450+0000","last_peered":"2024-03-16T12:20:37.582450+0000","last_clean":"2024-03-16T12:20:37.582450+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:20:37.582450+0000","last_undegraded":"2024-03-16T12:20:37.582450+0000","last_fullsized":"2024-03-16T12:20:37.582450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774617+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:20:12.774617+0000","last_peered":"2024-03-16T12:20:12.774617+0000","last_clean":"2024-03-16T12:20:12.774617+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:20:12.774617+0000","last_undegraded":"2024-03-16T12:20:12.774617+0000","last_fullsized":"2024-03-16T12:20:12.774617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547967+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:19:44.547967+0000","last_peered":"2024-03-16T12:19:44.547967+0000","last_clean":"2024-03-16T12:19:44.547967+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:19:44.547967+0000","last_undegraded":"2024-03-16T12:19:44.547967+0000","last_fullsized":"2024-03-16T12:19:44.547967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547367+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:19:44.547367+0000","last_peered":"2024-03-16T12:19:44.547367+0000","last_clean":"2024-03-16T12:19:44.547367+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:19:44.547367+0000","last_undegraded":"2024-03-16T12:19:44.547367+0000","last_fullsized":"2024-03-16T12:19:44.547367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792543+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:20:12.792543+0000","last_peered":"2024-03-16T12:20:12.792543+0000","last_clean":"2024-03-16T12:20:12.792543+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:20:12.792543+0000","last_undegraded":"2024-03-16T12:20:12.792543+0000","last_fullsized":"2024-03-16T12:20:12.792543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545566+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:19:44.545566+0000","last_peered":"2024-03-16T12:19:44.545566+0000","last_clean":"2024-03-16T12:19:44.545566+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:19:44.545566+0000","last_undegraded":"2024-03-16T12:19:44.545566+0000","last_fullsized":"2024-03-16T12:19:44.545566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555845+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:44.555845+0000","last_peered":"2024-03-16T12:19:44.555845+0000","last_clean":"2024-03-16T12:19:44.555845+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:44.555845+0000","last_undegraded":"2024-03-16T12:19:44.555845+0000","last_fullsized":"2024-03-16T12:19:44.555845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792861+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:20:12.792861+0000","last_peered":"2024-03-16T12:20:12.792861+0000","last_clean":"2024-03-16T12:20:12.792861+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:20:12.792861+0000","last_undegraded":"2024-03-16T12:20:12.792861+0000","last_fullsized":"2024-03-16T12:20:12.792861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172253+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:19:45.172253+0000","last_peered":"2024-03-16T12:19:45.172253+0000","last_clean":"2024-03-16T12:19:45.172253+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:19:45.172253+0000","last_undegraded":"2024-03-16T12:19:45.172253+0000","last_fullsized":"2024-03-16T12:19:45.172253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549520+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:19:44.549520+0000","last_peered":"2024-03-16T12:19:44.549520+0000","last_clean":"2024-03-16T12:19:44.549520+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:19:44.549520+0000","last_undegraded":"2024-03-16T12:19:44.549520+0000","last_fullsized":"2024-03-16T12:19:44.549520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.801683+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:20:12.801683+0000","last_peered":"2024-03-16T12:20:12.801683+0000","last_clean":"2024-03-16T12:20:12.801683+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:20:12.801683+0000","last_undegraded":"2024-03-16T12:20:12.801683+0000","last_fullsized":"2024-03-16T12:20:12.801683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546386+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:19:44.546386+0000","last_peered":"2024-03-16T12:19:44.546386+0000","last_clean":"2024-03-16T12:19:44.546386+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:19:44.546386+0000","last_undegraded":"2024-03-16T12:19:44.546386+0000","last_fullsized":"2024-03-16T12:19:44.546386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:35.579944+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:20:35.579944+0000","last_peered":"2024-03-16T12:20:35.579944+0000","last_clean":"2024-03-16T12:20:35.579944+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:20:35.579944+0000","last_undegraded":"2024-03-16T12:20:35.579944+0000","last_fullsized":"2024-03-16T12:20:35.579944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787246+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:20:12.787246+0000","last_peered":"2024-03-16T12:20:12.787246+0000","last_clean":"2024-03-16T12:20:12.787246+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:20:12.787246+0000","last_undegraded":"2024-03-16T12:20:12.787246+0000","last_fullsized":"2024-03-16T12:20:12.787246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845520+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:19:44.845520+0000","last_peered":"2024-03-16T12:19:44.845520+0000","last_clean":"2024-03-16T12:19:44.845520+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:19:44.845520+0000","last_undegraded":"2024-03-16T12:19:44.845520+0000","last_fullsized":"2024-03-16T12:19:44.845520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548079+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:44.548079+0000","last_peered":"2024-03-16T12:19:44.548079+0000","last_clean":"2024-03-16T12:19:44.548079+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:44.548079+0000","last_undegraded":"2024-03-16T12:19:44.548079+0000","last_fullsized":"2024-03-16T12:19:44.548079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794077+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:20:12.794077+0000","last_peered":"2024-03-16T12:20:12.794077+0000","last_clean":"2024-03-16T12:20:12.794077+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:20:12.794077+0000","last_undegraded":"2024-03-16T12:20:12.794077+0000","last_fullsized":"2024-03-16T12:20:12.794077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546550+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:19:44.546550+0000","last_peered":"2024-03-16T12:19:44.546550+0000","last_clean":"2024-03-16T12:19:44.546550+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:19:44.546550+0000","last_undegraded":"2024-03-16T12:19:44.546550+0000","last_fullsized":"2024-03-16T12:19:44.546550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549429+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:44.549429+0000","last_peered":"2024-03-16T12:19:44.549429+0000","last_clean":"2024-03-16T12:19:44.549429+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:44.549429+0000","last_undegraded":"2024-03-16T12:19:44.549429+0000","last_fullsized":"2024-03-16T12:19:44.549429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785764+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:20:12.785764+0000","last_peered":"2024-03-16T12:20:12.785764+0000","last_clean":"2024-03-16T12:20:12.785764+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:20:12.785764+0000","last_undegraded":"2024-03-16T12:20:12.785764+0000","last_fullsized":"2024-03-16T12:20:12.785764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845603+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:19:44.845603+0000","last_peered":"2024-03-16T12:19:44.845603+0000","last_clean":"2024-03-16T12:19:44.845603+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:19:44.845603+0000","last_undegraded":"2024-03-16T12:19:44.845603+0000","last_fullsized":"2024-03-16T12:19:44.845603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":253,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:36.581055+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:20:36.581055+0000","last_peered":"2024-03-16T12:20:36.581055+0000","last_clean":"2024-03-16T12:20:36.581055+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:20:36.581055+0000","last_undegraded":"2024-03-16T12:20:36.581055+0000","last_fullsized":"2024-03-16T12:20:36.581055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.753013+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:20:12.753013+0000","last_peered":"2024-03-16T12:20:12.753013+0000","last_clean":"2024-03-16T12:20:12.753013+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:20:12.753013+0000","last_undegraded":"2024-03-16T12:20:12.753013+0000","last_fullsized":"2024-03-16T12:20:12.753013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550165+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:19:44.550165+0000","last_peered":"2024-03-16T12:19:44.550165+0000","last_clean":"2024-03-16T12:19:44.550165+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:19:44.550165+0000","last_undegraded":"2024-03-16T12:19:44.550165+0000","last_fullsized":"2024-03-16T12:19:44.550165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565318+0000","last_change":"2024-03-16T12:19:44.565318+0000","last_active":"2024-03-16T12:19:44.565318+0000","last_peered":"2024-03-16T12:19:44.565318+0000","last_clean":"2024-03-16T12:19:44.565318+0000","last_became_active":"2024-03-16T12:19:44.551024+0000","last_became_peered":"2024-03-16T12:19:44.551024+0000","last_unstale":"2024-03-16T12:19:44.565318+0000","last_undegraded":"2024-03-16T12:19:44.565318+0000","last_fullsized":"2024-03-16T12:19:44.565318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:15.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800239+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:20:12.800239+0000","last_peered":"2024-03-16T12:20:12.800239+0000","last_clean":"2024-03-16T12:20:12.800239+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:20:12.800239+0000","last_undegraded":"2024-03-16T12:20:12.800239+0000","last_fullsized":"2024-03-16T12:20:12.800239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.544965+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:19:44.544965+0000","last_peered":"2024-03-16T12:19:44.544965+0000","last_clean":"2024-03-16T12:19:44.544965+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:19:44.544965+0000","last_undegraded":"2024-03-16T12:19:44.544965+0000","last_fullsized":"2024-03-16T12:19:44.544965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549124+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:44.549124+0000","last_peered":"2024-03-16T12:19:44.549124+0000","last_clean":"2024-03-16T12:19:44.549124+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:44.549124+0000","last_undegraded":"2024-03-16T12:19:44.549124+0000","last_fullsized":"2024-03-16T12:19:44.549124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790226+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:20:12.790226+0000","last_peered":"2024-03-16T12:20:12.790226+0000","last_clean":"2024-03-16T12:20:12.790226+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:20:12.790226+0000","last_undegraded":"2024-03-16T12:20:12.790226+0000","last_fullsized":"2024-03-16T12:20:12.790226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.846915+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:19:44.846915+0000","last_peered":"2024-03-16T12:19:44.846915+0000","last_clean":"2024-03-16T12:19:44.846915+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:19:44.846915+0000","last_undegraded":"2024-03-16T12:19:44.846915+0000","last_fullsized":"2024-03-16T12:19:44.846915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:32.628969+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:20:32.628969+0000","last_peered":"2024-03-16T12:20:32.628969+0000","last_clean":"2024-03-16T12:20:32.628969+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:20:32.628969+0000","last_undegraded":"2024-03-16T12:20:32.628969+0000","last_fullsized":"2024-03-16T12:20:32.628969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.754234+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:20:12.754234+0000","last_peered":"2024-03-16T12:20:12.754234+0000","last_clean":"2024-03-16T12:20:12.754234+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:20:12.754234+0000","last_undegraded":"2024-03-16T12:20:12.754234+0000","last_fullsized":"2024-03-16T12:20:12.754234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565998+0000","last_change":"2024-03-16T12:19:44.565998+0000","last_active":"2024-03-16T12:19:44.565998+0000","last_peered":"2024-03-16T12:19:44.565998+0000","last_clean":"2024-03-16T12:19:44.565998+0000","last_became_active":"2024-03-16T12:19:44.564591+0000","last_became_peered":"2024-03-16T12:19:44.564591+0000","last_unstale":"2024-03-16T12:19:44.565998+0000","last_undegraded":"2024-03-16T12:19:44.565998+0000","last_fullsized":"2024-03-16T12:19:44.565998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:57:48.227439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543302+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:44.543302+0000","last_peered":"2024-03-16T12:19:44.543302+0000","last_clean":"2024-03-16T12:19:44.543302+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:44.543302+0000","last_undegraded":"2024-03-16T12:19:44.543302+0000","last_fullsized":"2024-03-16T12:19:44.543302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782528+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:20:12.782528+0000","last_peered":"2024-03-16T12:20:12.782528+0000","last_clean":"2024-03-16T12:20:12.782528+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:20:12.782528+0000","last_undegraded":"2024-03-16T12:20:12.782528+0000","last_fullsized":"2024-03-16T12:20:12.782528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548883+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:19:44.548883+0000","last_peered":"2024-03-16T12:19:44.548883+0000","last_clean":"2024-03-16T12:19:44.548883+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:19:44.548883+0000","last_undegraded":"2024-03-16T12:19:44.548883+0000","last_fullsized":"2024-03-16T12:19:44.548883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543827+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:44.543827+0000","last_peered":"2024-03-16T12:19:44.543827+0000","last_clean":"2024-03-16T12:19:44.543827+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:44.543827+0000","last_undegraded":"2024-03-16T12:19:44.543827+0000","last_fullsized":"2024-03-16T12:19:44.543827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.768313+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:20:12.768313+0000","last_peered":"2024-03-16T12:20:12.768313+0000","last_clean":"2024-03-16T12:20:12.768313+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:20:12.768313+0000","last_undegraded":"2024-03-16T12:20:12.768313+0000","last_fullsized":"2024-03-16T12:20:12.768313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543432+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:19:44.543432+0000","last_peered":"2024-03-16T12:19:44.543432+0000","last_clean":"2024-03-16T12:19:44.543432+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:19:44.543432+0000","last_undegraded":"2024-03-16T12:19:44.543432+0000","last_fullsized":"2024-03-16T12:19:44.543432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847386+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:19:44.847386+0000","last_peered":"2024-03-16T12:19:44.847386+0000","last_clean":"2024-03-16T12:19:44.847386+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:19:44.847386+0000","last_undegraded":"2024-03-16T12:19:44.847386+0000","last_fullsized":"2024-03-16T12:19:44.847386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793024+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:20:12.793024+0000","last_peered":"2024-03-16T12:20:12.793024+0000","last_clean":"2024-03-16T12:20:12.793024+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:20:12.793024+0000","last_undegraded":"2024-03-16T12:20:12.793024+0000","last_fullsized":"2024-03-16T12:20:12.793024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.557767+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:19:44.557767+0000","last_peered":"2024-03-16T12:19:44.557767+0000","last_clean":"2024-03-16T12:19:44.557767+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:19:44.557767+0000","last_undegraded":"2024-03-16T12:19:44.557767+0000","last_fullsized":"2024-03-16T12:19:44.557767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547765+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:19:44.547765+0000","last_peered":"2024-03-16T12:19:44.547765+0000","last_clean":"2024-03-16T12:19:44.547765+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:19:44.547765+0000","last_undegraded":"2024-03-16T12:19:44.547765+0000","last_fullsized":"2024-03-16T12:19:44.547765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786168+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:20:12.786168+0000","last_peered":"2024-03-16T12:20:12.786168+0000","last_clean":"2024-03-16T12:20:12.786168+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:20:12.786168+0000","last_undegraded":"2024-03-16T12:20:12.786168+0000","last_fullsized":"2024-03-16T12:20:12.786168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559339+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:19:44.559339+0000","last_peered":"2024-03-16T12:19:44.559339+0000","last_clean":"2024-03-16T12:19:44.559339+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:19:44.559339+0000","last_undegraded":"2024-03-16T12:19:44.559339+0000","last_fullsized":"2024-03-16T12:19:44.559339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776386+0000","last_change":"2024-03-16T12:19:44.552241+0000","last_active":"2024-03-16T12:20:12.776386+0000","last_peered":"2024-03-16T12:20:12.776386+0000","last_clean":"2024-03-16T12:20:12.776386+0000","last_became_active":"2024-03-16T12:19:44.551762+0000","last_became_peered":"2024-03-16T12:19:44.551762+0000","last_unstale":"2024-03-16T12:20:12.776386+0000","last_undegraded":"2024-03-16T12:20:12.776386+0000","last_fullsized":"2024-03-16T12:20:12.776386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:24.454587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:32.628398+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:20:32.628398+0000","last_peered":"2024-03-16T12:20:32.628398+0000","last_clean":"2024-03-16T12:20:32.628398+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:20:32.628398+0000","last_undegraded":"2024-03-16T12:20:32.628398+0000","last_fullsized":"2024-03-16T12:20:32.628398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547939+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:19:44.547939+0000","last_peered":"2024-03-16T12:19:44.547939+0000","last_clean":"2024-03-16T12:19:44.547939+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:19:44.547939+0000","last_undegraded":"2024-03-16T12:19:44.547939+0000","last_fullsized":"2024-03-16T12:19:44.547939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790228+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:20:12.790228+0000","last_peered":"2024-03-16T12:20:12.790228+0000","last_clean":"2024-03-16T12:20:12.790228+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:20:12.790228+0000","last_undegraded":"2024-03-16T12:20:12.790228+0000","last_fullsized":"2024-03-16T12:20:12.790228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546475+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:44.546475+0000","last_peered":"2024-03-16T12:19:44.546475+0000","last_clean":"2024-03-16T12:19:44.546475+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:44.546475+0000","last_undegraded":"2024-03-16T12:19:44.546475+0000","last_fullsized":"2024-03-16T12:19:44.546475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548608+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:19:44.548608+0000","last_peered":"2024-03-16T12:19:44.548608+0000","last_clean":"2024-03-16T12:19:44.548608+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:19:44.548608+0000","last_undegraded":"2024-03-16T12:19:44.548608+0000","last_fullsized":"2024-03-16T12:19:44.548608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.566987+0000","last_change":"2024-03-16T12:19:44.566987+0000","last_active":"2024-03-16T12:19:44.566987+0000","last_peered":"2024-03-16T12:19:44.566987+0000","last_clean":"2024-03-16T12:19:44.566987+0000","last_became_active":"2024-03-16T12:19:44.566529+0000","last_became_peered":"2024-03-16T12:19:44.566529+0000","last_unstale":"2024-03-16T12:19:44.566987+0000","last_undegraded":"2024-03-16T12:19:44.566987+0000","last_fullsized":"2024-03-16T12:19:44.566987+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:53:03.124434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785242+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:20:12.785242+0000","last_peered":"2024-03-16T12:20:12.785242+0000","last_clean":"2024-03-16T12:20:12.785242+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:20:12.785242+0000","last_undegraded":"2024-03-16T12:20:12.785242+0000","last_fullsized":"2024-03-16T12:20:12.785242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548660+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:44.548660+0000","last_peered":"2024-03-16T12:19:44.548660+0000","last_clean":"2024-03-16T12:19:44.548660+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:44.548660+0000","last_undegraded":"2024-03-16T12:19:44.548660+0000","last_fullsized":"2024-03-16T12:19:44.548660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559544+0000","last_change":"2024-03-16T12:19:44.559544+0000","last_active":"2024-03-16T12:19:44.559544+0000","last_peered":"2024-03-16T12:19:44.559544+0000","last_clean":"2024-03-16T12:19:44.559544+0000","last_became_active":"2024-03-16T12:19:44.559030+0000","last_became_peered":"2024-03-16T12:19:44.559030+0000","last_unstale":"2024-03-16T12:19:44.559544+0000","last_undegraded":"2024-03-16T12:19:44.559544+0000","last_fullsized":"2024-03-16T12:19:44.559544+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:46.496512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7,6],"acting":[11,10,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543688+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:44.543688+0000","last_peered":"2024-03-16T12:19:44.543688+0000","last_clean":"2024-03-16T12:19:44.543688+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:44.543688+0000","last_undegraded":"2024-03-16T12:19:44.543688+0000","last_fullsized":"2024-03-16T12:19:44.543688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786161+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:20:12.786161+0000","last_peered":"2024-03-16T12:20:12.786161+0000","last_clean":"2024-03-16T12:20:12.786161+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:20:12.786161+0000","last_undegraded":"2024-03-16T12:20:12.786161+0000","last_fullsized":"2024-03-16T12:20:12.786161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546367+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:19:44.546367+0000","last_peered":"2024-03-16T12:19:44.546367+0000","last_clean":"2024-03-16T12:19:44.546367+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:19:44.546367+0000","last_undegraded":"2024-03-16T12:19:44.546367+0000","last_fullsized":"2024-03-16T12:19:44.546367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847757+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:19:44.847757+0000","last_peered":"2024-03-16T12:19:44.847757+0000","last_clean":"2024-03-16T12:19:44.847757+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:19:44.847757+0000","last_undegraded":"2024-03-16T12:19:44.847757+0000","last_fullsized":"2024-03-16T12:19:44.847757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.783807+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:20:12.783807+0000","last_peered":"2024-03-16T12:20:12.783807+0000","last_clean":"2024-03-16T12:20:12.783807+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:20:12.783807+0000","last_undegraded":"2024-03-16T12:20:12.783807+0000","last_fullsized":"2024-03-16T12:20:12.783807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546775+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:19:44.546775+0000","last_peered":"2024-03-16T12:19:44.546775+0000","last_clean":"2024-03-16T12:19:44.546775+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:19:44.546775+0000","last_undegraded":"2024-03-16T12:19:44.546775+0000","last_fullsized":"2024-03-16T12:19:44.546775+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546916+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:44.546916+0000","last_peered":"2024-03-16T12:19:44.546916+0000","last_clean":"2024-03-16T12:19:44.546916+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:44.546916+0000","last_undegraded":"2024-03-16T12:19:44.546916+0000","last_fullsized":"2024-03-16T12:19:44.546916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804835+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:20:12.804835+0000","last_peered":"2024-03-16T12:20:12.804835+0000","last_clean":"2024-03-16T12:20:12.804835+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:20:12.804835+0000","last_undegraded":"2024-03-16T12:20:12.804835+0000","last_fullsized":"2024-03-16T12:20:12.804835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546439+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:19:44.546439+0000","last_peered":"2024-03-16T12:19:44.546439+0000","last_clean":"2024-03-16T12:19:44.546439+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:19:44.546439+0000","last_undegraded":"2024-03-16T12:19:44.546439+0000","last_fullsized":"2024-03-16T12:19:44.546439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168814+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:45.168814+0000","last_peered":"2024-03-16T12:19:45.168814+0000","last_clean":"2024-03-16T12:19:45.168814+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:45.168814+0000","last_undegraded":"2024-03-16T12:19:45.168814+0000","last_fullsized":"2024-03-16T12:19:45.168814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.740265+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:20:12.740265+0000","last_peered":"2024-03-16T12:20:12.740265+0000","last_clean":"2024-03-16T12:20:12.740265+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:20:12.740265+0000","last_undegraded":"2024-03-16T12:20:12.740265+0000","last_fullsized":"2024-03-16T12:20:12.740265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847128+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:19:44.847128+0000","last_peered":"2024-03-16T12:19:44.847128+0000","last_clean":"2024-03-16T12:19:44.847128+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:19:44.847128+0000","last_undegraded":"2024-03-16T12:19:44.847128+0000","last_fullsized":"2024-03-16T12:19:44.847128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546423+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:44.546423+0000","last_peered":"2024-03-16T12:19:44.546423+0000","last_clean":"2024-03-16T12:19:44.546423+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:44.546423+0000","last_undegraded":"2024-03-16T12:19:44.546423+0000","last_fullsized":"2024-03-16T12:19:44.546423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803410+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:20:12.803410+0000","last_peered":"2024-03-16T12:20:12.803410+0000","last_clean":"2024-03-16T12:20:12.803410+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:20:12.803410+0000","last_undegraded":"2024-03-16T12:20:12.803410+0000","last_fullsized":"2024-03-16T12:20:12.803410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546286+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:19:44.546286+0000","last_peered":"2024-03-16T12:19:44.546286+0000","last_clean":"2024-03-16T12:19:44.546286+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:19:44.546286+0000","last_undegraded":"2024-03-16T12:19:44.546286+0000","last_fullsized":"2024-03-16T12:19:44.546286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547011+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:19:44.547011+0000","last_peered":"2024-03-16T12:19:44.547011+0000","last_clean":"2024-03-16T12:19:44.547011+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:19:44.547011+0000","last_undegraded":"2024-03-16T12:19:44.547011+0000","last_fullsized":"2024-03-16T12:19:44.547011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804622+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:20:12.804622+0000","last_peered":"2024-03-16T12:20:12.804622+0000","last_clean":"2024-03-16T12:20:12.804622+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:20:12.804622+0000","last_undegraded":"2024-03-16T12:20:12.804622+0000","last_fullsized":"2024-03-16T12:20:12.804622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546286+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:19:44.546286+0000","last_peered":"2024-03-16T12:19:44.546286+0000","last_clean":"2024-03-16T12:19:44.546286+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:19:44.546286+0000","last_undegraded":"2024-03-16T12:19:44.546286+0000","last_fullsized":"2024-03-16T12:19:44.546286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555974+0000","last_change":"2024-03-16T12:19:44.555974+0000","last_active":"2024-03-16T12:19:44.555974+0000","last_peered":"2024-03-16T12:19:44.555974+0000","last_clean":"2024-03-16T12:19:44.555974+0000","last_became_active":"2024-03-16T12:19:44.550254+0000","last_became_peered":"2024-03-16T12:19:44.550254+0000","last_unstale":"2024-03-16T12:19:44.555974+0000","last_undegraded":"2024-03-16T12:19:44.555974+0000","last_fullsized":"2024-03-16T12:19:44.555974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789419+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:20:12.789419+0000","last_peered":"2024-03-16T12:20:12.789419+0000","last_clean":"2024-03-16T12:20:12.789419+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:20:12.789419+0000","last_undegraded":"2024-03-16T12:20:12.789419+0000","last_fullsized":"2024-03-16T12:20:12.789419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168826+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:19:45.168826+0000","last_peered":"2024-03-16T12:19:45.168826+0000","last_clean":"2024-03-16T12:19:45.168826+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:19:45.168826+0000","last_undegraded":"2024-03-16T12:19:45.168826+0000","last_fullsized":"2024-03-16T12:19:45.168826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548123+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:19:44.548123+0000","last_peered":"2024-03-16T12:19:44.548123+0000","last_clean":"2024-03-16T12:19:44.548123+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:19:44.548123+0000","last_undegraded":"2024-03-16T12:19:44.548123+0000","last_fullsized":"2024-03-16T12:19:44.548123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786614+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:20:12.786614+0000","last_peered":"2024-03-16T12:20:12.786614+0000","last_clean":"2024-03-16T12:20:12.786614+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:20:12.786614+0000","last_undegraded":"2024-03-16T12:20:12.786614+0000","last_fullsized":"2024-03-16T12:20:12.786614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.171930+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:19:45.171930+0000","last_peered":"2024-03-16T12:19:45.171930+0000","last_clean":"2024-03-16T12:19:45.171930+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:19:45.171930+0000","last_undegraded":"2024-03-16T12:19:45.171930+0000","last_fullsized":"2024-03-16T12:19:45.171930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549942+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:19:44.549942+0000","last_peered":"2024-03-16T12:19:44.549942+0000","last_clean":"2024-03-16T12:19:44.549942+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:19:44.549942+0000","last_undegraded":"2024-03-16T12:19:44.549942+0000","last_fullsized":"2024-03-16T12:19:44.549942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788679+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:20:12.788679+0000","last_peered":"2024-03-16T12:20:12.788679+0000","last_clean":"2024-03-16T12:20:12.788679+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:20:12.788679+0000","last_undegraded":"2024-03-16T12:20:12.788679+0000","last_fullsized":"2024-03-16T12:20:12.788679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549223+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:19:44.549223+0000","last_peered":"2024-03-16T12:19:44.549223+0000","last_clean":"2024-03-16T12:19:44.549223+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:19:44.549223+0000","last_undegraded":"2024-03-16T12:19:44.549223+0000","last_fullsized":"2024-03-16T12:19:44.549223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559443+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:19:44.559443+0000","last_peered":"2024-03-16T12:19:44.559443+0000","last_clean":"2024-03-16T12:19:44.559443+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:19:44.559443+0000","last_undegraded":"2024-03-16T12:19:44.559443+0000","last_fullsized":"2024-03-16T12:19:44.559443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.771593+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:20:12.771593+0000","last_peered":"2024-03-16T12:20:12.771593+0000","last_clean":"2024-03-16T12:20:12.771593+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:20:12.771593+0000","last_undegraded":"2024-03-16T12:20:12.771593+0000","last_fullsized":"2024-03-16T12:20:12.771593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774110+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:20:12.774110+0000","last_peered":"2024-03-16T12:20:12.774110+0000","last_clean":"2024-03-16T12:20:12.774110+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:20:12.774110+0000","last_undegraded":"2024-03-16T12:20:12.774110+0000","last_fullsized":"2024-03-16T12:20:12.774110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.780975+0000","last_change":"2024-03-16T12:19:44.847899+0000","last_active":"2024-03-16T12:20:12.780975+0000","last_peered":"2024-03-16T12:20:12.780975+0000","last_clean":"2024-03-16T12:20:12.780975+0000","last_became_active":"2024-03-16T12:19:44.847589+0000","last_became_peered":"2024-03-16T12:19:44.847589+0000","last_unstale":"2024-03-16T12:20:12.780975+0000","last_undegraded":"2024-03-16T12:20:12.780975+0000","last_fullsized":"2024-03-16T12:20:12.780975+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:59.091516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.802745+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:20:12.802745+0000","last_peered":"2024-03-16T12:20:12.802745+0000","last_clean":"2024-03-16T12:20:12.802745+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:20:12.802745+0000","last_undegraded":"2024-03-16T12:20:12.802745+0000","last_fullsized":"2024-03-16T12:20:12.802745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804406+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:20:12.804406+0000","last_peered":"2024-03-16T12:20:12.804406+0000","last_clean":"2024-03-16T12:20:12.804406+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:20:12.804406+0000","last_undegraded":"2024-03-16T12:20:12.804406+0000","last_fullsized":"2024-03-16T12:20:12.804406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804088+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:20:12.804088+0000","last_peered":"2024-03-16T12:20:12.804088+0000","last_clean":"2024-03-16T12:20:12.804088+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:20:12.804088+0000","last_undegraded":"2024-03-16T12:20:12.804088+0000","last_fullsized":"2024-03-16T12:20:12.804088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770383+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:20:12.770383+0000","last_peered":"2024-03-16T12:20:12.770383+0000","last_clean":"2024-03-16T12:20:12.770383+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:20:12.770383+0000","last_undegraded":"2024-03-16T12:20:12.770383+0000","last_fullsized":"2024-03-16T12:20:12.770383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776817+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:20:12.776817+0000","last_peered":"2024-03-16T12:20:12.776817+0000","last_clean":"2024-03-16T12:20:12.776817+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:20:12.776817+0000","last_undegraded":"2024-03-16T12:20:12.776817+0000","last_fullsized":"2024-03-16T12:20:12.776817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776082+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:20:12.776082+0000","last_peered":"2024-03-16T12:20:12.776082+0000","last_clean":"2024-03-16T12:20:12.776082+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:20:12.776082+0000","last_undegraded":"2024-03-16T12:20:12.776082+0000","last_fullsized":"2024-03-16T12:20:12.776082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799884+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:20:12.799884+0000","last_peered":"2024-03-16T12:20:12.799884+0000","last_clean":"2024-03-16T12:20:12.799884+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:20:12.799884+0000","last_undegraded":"2024-03-16T12:20:12.799884+0000","last_fullsized":"2024-03-16T12:20:12.799884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.760626+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:20:12.760626+0000","last_peered":"2024-03-16T12:20:12.760626+0000","last_clean":"2024-03-16T12:20:12.760626+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:20:12.760626+0000","last_undegraded":"2024-03-16T12:20:12.760626+0000","last_fullsized":"2024-03-16T12:20:12.760626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800202+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:20:12.800202+0000","last_peered":"2024-03-16T12:20:12.800202+0000","last_clean":"2024-03-16T12:20:12.800202+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:20:12.800202+0000","last_undegraded":"2024-03-16T12:20:12.800202+0000","last_fullsized":"2024-03-16T12:20:12.800202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.806499+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:20:12.806499+0000","last_peered":"2024-03-16T12:20:12.806499+0000","last_clean":"2024-03-16T12:20:12.806499+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:20:12.806499+0000","last_undegraded":"2024-03-16T12:20:12.806499+0000","last_fullsized":"2024-03-16T12:20:12.806499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781457+0000","last_change":"2024-03-16T12:19:44.849288+0000","last_active":"2024-03-16T12:20:12.781457+0000","last_peered":"2024-03-16T12:20:12.781457+0000","last_clean":"2024-03-16T12:20:12.781457+0000","last_became_active":"2024-03-16T12:19:44.848869+0000","last_became_peered":"2024-03-16T12:19:44.848869+0000","last_unstale":"2024-03-16T12:20:12.781457+0000","last_undegraded":"2024-03-16T12:20:12.781457+0000","last_fullsized":"2024-03-16T12:20:12.781457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:36.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66192,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":561,"num_read_kb":348,"num_write":169,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":198576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":500849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28692,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716460,"statfs":{"total":95995035648,"available":95965655040,"internally_reserved":0,"allocated":1716224,"data_stored":1150665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":500971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1096489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":500877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1092077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:20:40.710 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T12:20:40.713 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T12:20:40.713 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T12:20:40.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T12:20:41.196 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:41.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:40 smithi098 ceph-mon[31633]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:41.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:40 smithi153 ceph-mon[36901]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:41.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:40 smithi173 ceph-mon[36632]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:20:42.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:42 smithi098 ceph-mon[31633]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:42.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:42 smithi153 ceph-mon[36901]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:42.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:42 smithi173 ceph-mon[36632]: pgmap v380: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 -- 172.21.15.98:0/3800105918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c81074f0 msgr2=0x7fb5c810f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 --2- 172.21.15.98:0/3800105918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c81074f0 0x7fb5c810f0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00099c0 tx=0x7fb5c0009cd0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 -- 172.21.15.98:0/3800105918 shutdown_connections 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 --2- 172.21.15.98:0/3800105918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c81074f0 0x7fb5c810f0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 --2- 172.21.15.98:0/3800105918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c8102410 0x7fb5c81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 --2- 172.21.15.98:0/3800105918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 0x7fb5c8106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 -- 172.21.15.98:0/3800105918 >> 172.21.15.98:0/3800105918 conn(0x7fb5c80fdcc0 msgr2=0x7fb5c81000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 -- 172.21.15.98:0/3800105918 shutdown_connections 2024-03-16T12:20:43.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.524+0000 7fb5cee00700 1 -- 172.21.15.98:0/3800105918 wait complete. 2024-03-16T12:20:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.525+0000 7fb5cee00700 1 Processor -- start 2024-03-16T12:20:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.525+0000 7fb5cee00700 1 -- start start 2024-03-16T12:20:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 0x7fb5c81a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c81074f0 0x7fb5c819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8112530 con 0x7fb5c81074f0 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81123b0 con 0x7fb5c8102dc0 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cee00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81126b0 con 0x7fb5c8102410 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57550/0 (socket says 172.21.15.98:57550) 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 -- 172.21.15.98:0/4272610490 learned_addr learned my addr 172.21.15.98:0/4272610490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cd000700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 0x7fb5c81a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5ce400700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c81074f0 0x7fb5c819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 -- 172.21.15.98:0/4272610490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 msgr2=0x7fb5c81a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 0x7fb5c81a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.526+0000 7fb5cda00700 1 -- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c81074f0 msgr2=0x7fb5c819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5cda00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c81074f0 0x7fb5c819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5cda00700 1 -- 172.21.15.98:0/4272610490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c819b0c0 con 0x7fb5c8102410 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5cda00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00ef10 tx=0x7fb5bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc00c9c0 con 0x7fb5c8102410 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5bc019070 con 0x7fb5c8102410 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c819b350 con 0x7fb5c8102410 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc007e60 con 0x7fb5c8102410 2024-03-16T12:20:43.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.527+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c81a7670 con 0x7fb5c8102410 2024-03-16T12:20:43.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.528+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c8068f90 con 0x7fb5c8102410 2024-03-16T12:20:43.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.530+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb5bc012050 con 0x7fb5c8102410 2024-03-16T12:20:43.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.531+0000 7fb5c6000700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 0x7fb5b00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:43.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.532+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb5bc0ced20 con 0x7fb5c8102410 2024-03-16T12:20:43.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.532+0000 7fb5cd000700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 0x7fb5b00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:43.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.532+0000 7fb5cd000700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 0x7fb5b00a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8101790 tx=0x7fb5b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:43.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.536+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb5bc010040 con 0x7fb5c8102410 2024-03-16T12:20:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.806+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb5c81a7840 con 0x7fb5c8102410 2024-03-16T12:20:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.806+0000 7fb5c6000700 1 -- 172.21.15.98:0/4272610490 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb5bc09ba00 con 0x7fb5c8102410 2024-03-16T12:20:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:43.787 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T12:20:43.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.809+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 msgr2=0x7fb5b00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:43.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.809+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 0x7fb5b00a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8101790 tx=0x7fb5b4009260 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.809+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 msgr2=0x7fb5c81a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.809+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00ef10 tx=0x7fb5bc00c3a0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 shutdown_connections 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5c81074f0 0x7fb5c819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c8102410 0x7fb5c81a07a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb5b00a2910 0x7fb5b00a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 --2- 172.21.15.98:0/4272610490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c8102dc0 0x7fb5c81a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 >> 172.21.15.98:0/4272610490 conn(0x7fb5c80fdcc0 msgr2=0x7fb5c810ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 shutdown_connections 2024-03-16T12:20:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:43.810+0000 7fb5cee00700 1 -- 172.21.15.98:0/4272610490 wait complete. 2024-03-16T12:20:44.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:44 smithi098 ceph-mon[31633]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:44.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:44 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4272610490' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T12:20:44.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:44 smithi153 ceph-mon[36901]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:44.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:44 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4272610490' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T12:20:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:44 smithi173 ceph-mon[36632]: pgmap v381: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:44.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:44 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4272610490' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T12:20:44.609 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T12:20:44.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:20:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 -- 172.21.15.98:0/1202779233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a81099b0 msgr2=0x7f53a8110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 --2- 172.21.15.98:0/1202779233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a81099b0 0x7f53a8110240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ca0 tx=0x7f53a002c100 comp rx=0 tx=0).stop 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 -- 172.21.15.98:0/1202779233 shutdown_connections 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 --2- 172.21.15.98:0/1202779233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a81099b0 0x7f53a8110240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 --2- 172.21.15.98:0/1202779233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a8108ff0 0x7f53a8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 --2- 172.21.15.98:0/1202779233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8107e00 0x7f53a8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 -- 172.21.15.98:0/1202779233 >> 172.21.15.98:0/1202779233 conn(0x7f53a8076dc0 msgr2=0x7f53a80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.848+0000 7f53aee00640 1 -- 172.21.15.98:0/1202779233 shutdown_connections 2024-03-16T12:20:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.849+0000 7f53aee00640 1 -- 172.21.15.98:0/1202779233 wait complete. 2024-03-16T12:20:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 Processor -- start 2024-03-16T12:20:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 -- start start 2024-03-16T12:20:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8108ff0 0x7f53a819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a81099b0 0x7f53a81a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8113a60 con 0x7f53a8108ff0 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81138e0 con 0x7f53a8107e00 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53aee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8113be0 con 0x7f53a81099b0 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.850+0000 7f53a7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33332/0 (socket says 172.21.15.98:33332) 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 -- 172.21.15.98:0/648508775 learned_addr learned my addr 172.21.15.98:0/648508775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f539f400640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8108ff0 0x7f53a819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53ace00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a81099b0 0x7f53a81a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 -- 172.21.15.98:0/648508775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a81099b0 msgr2=0x7f53a81a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a81099b0 0x7f53a81a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 -- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8108ff0 msgr2=0x7f53a819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8108ff0 0x7f53a819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 -- 172.21.15.98:0/648508775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a81a5c60 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a7e00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f539800e9c0 tx=0x7f539800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800cd80 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53980035f0 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.851+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a81a5f50 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.852+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398010690 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.852+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81a63b0 con 0x7f53a8107e00 2024-03-16T12:20:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.853+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a810f0c0 con 0x7f53a8107e00 2024-03-16T12:20:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.855+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f53980037a0 con 0x7f53a8107e00 2024-03-16T12:20:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.856+0000 7f53a5600640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 0x7f538c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.856+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f53980cf150 con 0x7f53a8107e00 2024-03-16T12:20:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.856+0000 7f539f400640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 0x7f538c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.857+0000 7f539f400640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 0x7f538c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f53900095f0 tx=0x7f5390005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:44.863+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f539809be90 con 0x7f53a8107e00 2024-03-16T12:20:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.063+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53a802d3a0 con 0x7f53a8107e00 2024-03-16T12:20:45.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.065+0000 7f53a5600640 1 -- 172.21.15.98:0/648508775 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f539809b790 con 0x7f53a8107e00 2024-03-16T12:20:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:45.046 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 msgr2=0x7f538c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 0x7f538c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f53900095f0 tx=0x7f5390005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 msgr2=0x7f53a819e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f539800e9c0 tx=0x7f539800ee80 comp rx=0 tx=0).stop 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 shutdown_connections 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f538c0a2800 0x7f538c0a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53a81099b0 0x7f53a81a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53a8108ff0 0x7f53a819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 --2- 172.21.15.98:0/648508775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8107e00 0x7f53a819e040 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 >> 172.21.15.98:0/648508775 conn(0x7f53a8076dc0 msgr2=0x7f53a810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:45.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 shutdown_connections 2024-03-16T12:20:45.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.068+0000 7f53aee00640 1 -- 172.21.15.98:0/648508775 wait complete. 2024-03-16T12:20:45.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:20:45.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:20:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 -- 172.21.15.98:0/2613491438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 msgr2=0x7f12a4110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 --2- 172.21.15.98:0/2613491438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a4110240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f129c009be0 tx=0x7f129c02bef0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 -- 172.21.15.98:0/2613491438 shutdown_connections 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 --2- 172.21.15.98:0/2613491438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a4110240 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 --2- 172.21.15.98:0/2613491438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 0x7f12a4109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 --2- 172.21.15.98:0/2613491438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a4108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 -- 172.21.15.98:0/2613491438 >> 172.21.15.98:0/2613491438 conn(0x7f12a4076da0 msgr2=0x7f12a40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 -- 172.21.15.98:0/2613491438 shutdown_connections 2024-03-16T12:20:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.299+0000 7f12a8c00640 1 -- 172.21.15.98:0/2613491438 wait complete. 2024-03-16T12:20:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.300+0000 7f12a8c00640 1 Processor -- start 2024-03-16T12:20:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.300+0000 7f12a8c00640 1 -- start start 2024-03-16T12:20:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 0x7f12a419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a41a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4113ab0 con 0x7f12a4108ff0 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4113930 con 0x7f12a4107e00 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a8c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4113c30 con 0x7f12a41099b0 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33338/0 (socket says 172.21.15.98:33338) 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 -- 172.21.15.98:0/1615535125 learned_addr learned my addr 172.21.15.98:0/1615535125 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a41a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f129be00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 0x7f12a419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 -- 172.21.15.98:0/1615535125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 msgr2=0x7f12a41a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a41a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 -- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 msgr2=0x7f12a419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 0x7f12a419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.301+0000 7f12a1e00640 1 -- 172.21.15.98:0/1615535125 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a41a5cb0 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f12a1e00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f129400e9a0 tx=0x7f129400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129400cd80 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1294010910 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a41a5fa0 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294018a30 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.302+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a41a6400 con 0x7f12a4107e00 2024-03-16T12:20:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.303+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a4068a20 con 0x7f12a4107e00 2024-03-16T12:20:45.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.305+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1294010ac0 con 0x7f12a4107e00 2024-03-16T12:20:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.306+0000 7f129aa00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 0x7f12880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.306+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12940cf180 con 0x7f12a4107e00 2024-03-16T12:20:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.306+0000 7f129be00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 0x7f12880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.307+0000 7f129be00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 0x7f12880a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12a419f5a0 tx=0x7f128c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:45.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.315+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1294053740 con 0x7f12a4107e00 2024-03-16T12:20:45.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.514+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12a40623e0 con 0x7f12a4107e00 2024-03-16T12:20:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.515+0000 7f129aa00640 1 -- 172.21.15.98:0/1615535125 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f129409be60 con 0x7f12a4107e00 2024-03-16T12:20:45.496 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:20:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 msgr2=0x7f12880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 0x7f12880a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12a419f5a0 tx=0x7f128c005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 msgr2=0x7f12a419e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f129400e9a0 tx=0x7f129400ee60 comp rx=0 tx=0).stop 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 shutdown_connections 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12880a2800 0x7f12880a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12a41099b0 0x7f12a41a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12a4108ff0 0x7f12a419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 --2- 172.21.15.98:0/1615535125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12a4107e00 0x7f12a419e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 >> 172.21.15.98:0/1615535125 conn(0x7f12a4076da0 msgr2=0x7f12a410c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.518+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 shutdown_connections 2024-03-16T12:20:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.519+0000 7f12a8c00640 1 -- 172.21.15.98:0/1615535125 wait complete. 2024-03-16T12:20:45.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:20:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 -- 172.21.15.98:0/3501417491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 msgr2=0x7fc914100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 --2- 172.21.15.98:0/3501417491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc914100e70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc900009860 tx=0x7fc90002bb70 comp rx=0 tx=0).stop 2024-03-16T12:20:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 -- 172.21.15.98:0/3501417491 shutdown_connections 2024-03-16T12:20:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 --2- 172.21.15.98:0/3501417491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141013b0 0x7fc914101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 --2- 172.21.15.98:0/3501417491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc914100e70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 --2- 172.21.15.98:0/3501417491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141000b0 0x7fc9141004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.752+0000 7fc918a00640 1 -- 172.21.15.98:0/3501417491 >> 172.21.15.98:0/3501417491 conn(0x7fc9140fbc70 msgr2=0x7fc9140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.753+0000 7fc918a00640 1 -- 172.21.15.98:0/3501417491 shutdown_connections 2024-03-16T12:20:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.753+0000 7fc918a00640 1 -- 172.21.15.98:0/3501417491 wait complete. 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.753+0000 7fc918a00640 1 Processor -- start 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 -- start start 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc91419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141013b0 0x7fc9141a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f620 con 0x7fc9141013b0 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f4a0 con 0x7fc914100a10 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc918a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f7a0 con 0x7fc9141000b0 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc911e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc912800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141013b0 0x7fc9141a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc911400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc91419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc911e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57608/0 (socket says 172.21.15.98:57608) 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.754+0000 7fc911e00640 1 -- 172.21.15.98:0/2098839334 learned_addr learned my addr 172.21.15.98:0/2098839334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 -- 172.21.15.98:0/2098839334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 msgr2=0x7fc91419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc91419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 -- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141013b0 msgr2=0x7fc9141a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141013b0 0x7fc9141a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 -- 172.21.15.98:0/2098839334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9141a1960 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc911e00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc90400ebc0 tx=0x7fc90400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904010630 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9141a1c50 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc904010cb0 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904018bd0 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.755+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9141a21a0 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.756+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc91410b040 con 0x7fc9141000b0 2024-03-16T12:20:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.758+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc9040107d0 con 0x7fc9141000b0 2024-03-16T12:20:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.759+0000 7fc90aa00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 0x7fc8f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.760+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc9040cf490 con 0x7fc9141000b0 2024-03-16T12:20:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.760+0000 7fc911400640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 0x7fc8f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.760+0000 7fc911400640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 0x7fc8f80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc900009830 tx=0x7fc900034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.763+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc90409bb20 con 0x7fc9141000b0 2024-03-16T12:20:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.964+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc91402d1b0 con 0x7fc9141000b0 2024-03-16T12:20:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.964+0000 7fc90aa00640 1 -- 172.21.15.98:0/2098839334 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fc91402d1b0 con 0x7fc9141000b0 2024-03-16T12:20:45.945 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 msgr2=0x7fc8f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 0x7fc8f80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc900009830 tx=0x7fc900034ab0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 msgr2=0x7fc914199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc90400ebc0 tx=0x7fc90400c6a0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 shutdown_connections 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc8f80a2800 0x7fc8f80a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9141013b0 0x7fc9141a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc914100a10 0x7fc91419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 --2- 172.21.15.98:0/2098839334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9141000b0 0x7fc914199d50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:45.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 >> 172.21.15.98:0/2098839334 conn(0x7fc9140fbc70 msgr2=0x7fc9140fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:45.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 shutdown_connections 2024-03-16T12:20:45.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:45.967+0000 7fc918a00640 1 -- 172.21.15.98:0/2098839334 wait complete. 2024-03-16T12:20:45.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:20:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 -- 172.21.15.98:0/373992165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5630107e00 msgr2=0x7f5630108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 --2- 172.21.15.98:0/373992165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5630107e00 0x7f5630108220 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5620009900 tx=0x7f562002bc50 comp rx=0 tx=0).stop 2024-03-16T12:20:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 -- 172.21.15.98:0/373992165 shutdown_connections 2024-03-16T12:20:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 --2- 172.21.15.98:0/373992165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56301099b0 0x7f5630110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 --2- 172.21.15.98:0/373992165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 0x7f5630109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 --2- 172.21.15.98:0/373992165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5630107e00 0x7f5630108220 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 -- 172.21.15.98:0/373992165 >> 172.21.15.98:0/373992165 conn(0x7f5630076da0 msgr2=0x7f56300791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 -- 172.21.15.98:0/373992165 shutdown_connections 2024-03-16T12:20:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.202+0000 7f5636c00640 1 -- 172.21.15.98:0/373992165 wait complete. 2024-03-16T12:20:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 Processor -- start 2024-03-16T12:20:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 -- start start 2024-03-16T12:20:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 0x7f563019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 0x7f56301a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5630113a80 con 0x7f5630108ff0 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5630113900 con 0x7f5630107e00 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.203+0000 7f5636c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5630113c00 con 0x7f56301099b0 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33364/0 (socket says 172.21.15.98:33364) 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 -- 172.21.15.98:0/3243503137 learned_addr learned my addr 172.21.15.98:0/3243503137 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f5634c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 0x7f56301a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f5627400640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 0x7f563019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 -- 172.21.15.98:0/3243503137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 msgr2=0x7f56301a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 0x7f56301a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 -- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 msgr2=0x7f563019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 0x7f563019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 -- 172.21.15.98:0/3243503137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56301a5db0 con 0x7f5630107e00 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f5634c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 0x7f56301a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.204+0000 7f562fe00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5620000c00 tx=0x7f562002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.205+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620042020 con 0x7f5630107e00 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.205+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5620004450 con 0x7f5630107e00 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.205+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56301a6040 con 0x7f5630107e00 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.205+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562003e460 con 0x7f5630107e00 2024-03-16T12:20:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.205+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301a65e0 con 0x7f5630107e00 2024-03-16T12:20:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.206+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563010f120 con 0x7f5630107e00 2024-03-16T12:20:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.208+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5620045020 con 0x7f5630107e00 2024-03-16T12:20:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.209+0000 7f562d600640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.209+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f56200f0ca0 con 0x7f5630107e00 2024-03-16T12:20:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.209+0000 7f5627400640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.210+0000 7f5627400640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 0x7f56140a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f563019f6a0 tx=0x7f5618005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.213+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f562004c020 con 0x7f5630107e00 2024-03-16T12:20:46.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:46 smithi098 ceph-mon[31633]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:46.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:46 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/648508775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:46.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:46 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1615535125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:46 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2098839334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:46 smithi153 ceph-mon[36901]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:46.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:46 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/648508775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:46.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:46 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1615535125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:46 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2098839334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:46 smithi173 ceph-mon[36632]: pgmap v382: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:46.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:46 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/648508775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:46.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:46 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1615535125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:46 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2098839334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.414+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56300623e0 con 0x7f5630107e00 2024-03-16T12:20:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.414+0000 7f562d600640 1 -- 172.21.15.98:0/3243503137 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f56200bda70 con 0x7f5630107e00 2024-03-16T12:20:46.395 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:20:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.417+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 msgr2=0x7f56140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.417+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 0x7f56140a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f563019f6a0 tx=0x7f5618005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.417+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 msgr2=0x7f563019e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.417+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5620000c00 tx=0x7f562002c660 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 shutdown_connections 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f56140a2880 0x7f56140a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56301099b0 0x7f56301a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5630108ff0 0x7f563019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 --2- 172.21.15.98:0/3243503137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5630107e00 0x7f563019e190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 >> 172.21.15.98:0/3243503137 conn(0x7f5630076da0 msgr2=0x7f563010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 shutdown_connections 2024-03-16T12:20:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.418+0000 7f5636c00640 1 -- 172.21.15.98:0/3243503137 wait complete. 2024-03-16T12:20:46.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 -- 172.21.15.98:0/1177684467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 msgr2=0x7ff468074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 --2- 172.21.15.98:0/1177684467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468074640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff450009900 tx=0x7ff45002bcd0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 -- 172.21.15.98:0/1177684467 shutdown_connections 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 --2- 172.21.15.98:0/1177684467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff468109680 0x7ff46810bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 --2- 172.21.15.98:0/1177684467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468074640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 --2- 172.21.15.98:0/1177684467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4680757c0 0x7ff468073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 -- 172.21.15.98:0/1177684467 >> 172.21.15.98:0/1177684467 conn(0x7ff4680fcd80 msgr2=0x7ff4680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 -- 172.21.15.98:0/1177684467 shutdown_connections 2024-03-16T12:20:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.643+0000 7ff46e200640 1 -- 172.21.15.98:0/1177684467 wait complete. 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 Processor -- start 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 -- start start 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 0x7ff46819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468109680 0x7ff4681a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810f6c0 con 0x7ff468109680 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810f540 con 0x7ff4680741e0 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.644+0000 7ff46e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810f840 con 0x7ff4680757c0 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff467400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 0x7ff46819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff467400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33388/0 (socket says 172.21.15.98:33388) 2024-03-16T12:20:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff467e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468109680 0x7ff4681a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff467400640 1 -- 172.21.15.98:0/3431655350 learned_addr learned my addr 172.21.15.98:0/3431655350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 -- 172.21.15.98:0/3431655350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 msgr2=0x7ff468199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 -- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468109680 msgr2=0x7ff4681a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468109680 0x7ff4681a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.645+0000 7ff45fe00640 1 -- 172.21.15.98:0/3431655350 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4681a1930 con 0x7ff4680757c0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff45fe00640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 0x7ff46819a260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff450036040 tx=0x7ff45002c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450043020 con 0x7ff4680757c0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4500054e0 con 0x7ff4680757c0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4681a1bc0 con 0x7ff4680757c0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450003b50 con 0x7ff4680757c0 2024-03-16T12:20:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.646+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4681a20e0 con 0x7ff4680757c0 2024-03-16T12:20:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.647+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff468108440 con 0x7ff4680757c0 2024-03-16T12:20:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.650+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff450046020 con 0x7ff4680757c0 2024-03-16T12:20:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.652+0000 7ff464c00640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 0x7ff44c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.652+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff4500f0de0 con 0x7ff4680757c0 2024-03-16T12:20:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.652+0000 7ff467400640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 0x7ff44c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.653+0000 7ff467400640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 0x7ff44c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff46819b1f0 tx=0x7ff458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.656+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff45004d020 con 0x7ff4680757c0 2024-03-16T12:20:46.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.858+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff46802d3f0 con 0x7ff4680757c0 2024-03-16T12:20:46.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.858+0000 7ff464c00640 1 -- 172.21.15.98:0/3431655350 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7ff4500bdb20 con 0x7ff4680757c0 2024-03-16T12:20:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 msgr2=0x7ff44c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 0x7ff44c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff46819b1f0 tx=0x7ff458005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 msgr2=0x7ff46819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 0x7ff46819a260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff450036040 tx=0x7ff45002c6f0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 shutdown_connections 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff44c0a2800 0x7ff44c0a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff468109680 0x7ff4681a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680757c0 0x7ff46819a260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 --2- 172.21.15.98:0/3431655350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4680741e0 0x7ff468199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 >> 172.21.15.98:0/3431655350 conn(0x7ff4680fcd80 msgr2=0x7ff468105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 shutdown_connections 2024-03-16T12:20:46.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:46.862+0000 7ff46e200640 1 -- 172.21.15.98:0/3431655350 wait complete. 2024-03-16T12:20:46.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:20:47.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.077+0000 7f4edb000640 1 -- 172.21.15.98:0/3353199309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed4104bf0 msgr2=0x7f4ed4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.077+0000 7f4edb000640 1 --2- 172.21.15.98:0/3353199309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed4104bf0 0x7f4ed4105070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009900 tx=0x7f4ebc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 -- 172.21.15.98:0/3353199309 shutdown_connections 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 --2- 172.21.15.98:0/3353199309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed41055b0 0x7f4ed410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 --2- 172.21.15.98:0/3353199309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed4104bf0 0x7f4ed4105070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 --2- 172.21.15.98:0/3353199309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4103a00 0x7f4ed4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 -- 172.21.15.98:0/3353199309 >> 172.21.15.98:0/3353199309 conn(0x7f4ed40fef90 msgr2=0x7f4ed41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 -- 172.21.15.98:0/3353199309 shutdown_connections 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.078+0000 7f4edb000640 1 -- 172.21.15.98:0/3353199309 wait complete. 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 Processor -- start 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 -- start start 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4104bf0 0x7f4ed419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410f6d0 con 0x7f4ed4104bf0 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410f550 con 0x7f4ed41055b0 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4edb000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410f850 con 0x7f4ed4103a00 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54048/0 (socket says 172.21.15.98:54048) 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48710/0 (socket says 172.21.15.98:48710) 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed9000640 1 -- 172.21.15.98:0/512575343 learned_addr learned my addr 172.21.15.98:0/512575343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.079+0000 7f4ed3400640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4104bf0 0x7f4ed419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 -- 172.21.15.98:0/512575343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 msgr2=0x7f4ed4199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 -- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4104bf0 msgr2=0x7f4ed419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4104bf0 0x7f4ed419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 -- 172.21.15.98:0/512575343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed41a1910 con 0x7f4ed41055b0 2024-03-16T12:20:47.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:47.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed9000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00eb10 tx=0x7f4ecc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:47.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc004b20 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ecc004cc0 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc010760 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.080+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed41a1c00 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.081+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed41a2100 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.082+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed410acc0 con 0x7f4ed41055b0 2024-03-16T12:20:47.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.083+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4ecc005020 con 0x7f4ed41055b0 2024-03-16T12:20:47.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.085+0000 7f4ed0c00640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 0x7f4eb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.085+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4ecc0cf570 con 0x7f4ed41055b0 2024-03-16T12:20:47.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.085+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 0x7f4eb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.086+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 0x7f4eb80a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ed419b1d0 tx=0x7f4ec4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:47.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.093+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ecc09c250 con 0x7f4ed41055b0 2024-03-16T12:20:47.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:47 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3243503137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:47 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3431655350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:47 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3243503137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:47 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3431655350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.332+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ed402d1b0 con 0x7f4ed41055b0 2024-03-16T12:20:47.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.333+0000 7f4ed0c00640 1 -- 172.21.15.98:0/512575343 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7f4ecc09bb50 con 0x7f4ed41055b0 2024-03-16T12:20:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:47.314 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:20:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 msgr2=0x7f4eb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 0x7f4eb80a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4ed419b1d0 tx=0x7f4ec4005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 msgr2=0x7f4ed41a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00eb10 tx=0x7f4ecc00c2d0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 shutdown_connections 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4eb80a2800 0x7f4eb80a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed41055b0 0x7f4ed41a1270 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4104bf0 0x7f4ed419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 --2- 172.21.15.98:0/512575343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ed4103a00 0x7f4ed4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 >> 172.21.15.98:0/512575343 conn(0x7f4ed40fef90 msgr2=0x7f4ed4107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 shutdown_connections 2024-03-16T12:20:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.336+0000 7f4edb000640 1 -- 172.21.15.98:0/512575343 wait complete. 2024-03-16T12:20:47.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:47 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3243503137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:47 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3431655350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:47.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:20:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 -- 172.21.15.98:0/1390746411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c077d80 msgr2=0x7fa57c078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 --2- 172.21.15.98:0/1390746411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c077d80 0x7fa57c078620 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa574009ca0 tx=0x7fa57402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 -- 172.21.15.98:0/1390746411 shutdown_connections 2024-03-16T12:20:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 --2- 172.21.15.98:0/1390746411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c077d80 0x7fa57c078620 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 --2- 172.21.15.98:0/1390746411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c0773e0 0x7fa57c077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 --2- 172.21.15.98:0/1390746411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0761a0 0x7fa57c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 -- 172.21.15.98:0/1390746411 >> 172.21.15.98:0/1390746411 conn(0x7fa57c0fef90 msgr2=0x7fa57c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 -- 172.21.15.98:0/1390746411 shutdown_connections 2024-03-16T12:20:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.548+0000 7fa582600640 1 -- 172.21.15.98:0/1390746411 wait complete. 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.549+0000 7fa582600640 1 Processor -- start 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 -- start start 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 0x7fa57c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c077d80 0x7fa57c1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c113a90 con 0x7fa57c0773e0 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c113910 con 0x7fa57c0761a0 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa582600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c113c10 con 0x7fa57c077d80 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa57b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa57b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54074/0 (socket says 172.21.15.98:54074) 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa57b400640 1 -- 172.21.15.98:0/1425032700 learned_addr learned my addr 172.21.15.98:0/1425032700 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa57be00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c077d80 0x7fa57c1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.550+0000 7fa57aa00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 0x7fa57c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 -- 172.21.15.98:0/1425032700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c077d80 msgr2=0x7fa57c1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c077d80 0x7fa57c1a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 -- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 msgr2=0x7fa57c19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 0x7fa57c19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 -- 172.21.15.98:0/1425032700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c1a5c80 con 0x7fa57c0761a0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57aa00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 0x7fa57c19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa57b400640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e9f0 tx=0x7fa56c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00cd80 con 0x7fa57c0761a0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa56c010910 con 0x7fa57c0761a0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c018990 con 0x7fa57c0761a0 2024-03-16T12:20:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c1a5f70 con 0x7fa57c0761a0 2024-03-16T12:20:47.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.551+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c1a6380 con 0x7fa57c0761a0 2024-03-16T12:20:47.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.552+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57c1104b0 con 0x7fa57c0761a0 2024-03-16T12:20:47.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.555+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa56c010ac0 con 0x7fa57c0761a0 2024-03-16T12:20:47.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.557+0000 7fa573e00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 0x7fa5600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.557+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa56c0cf120 con 0x7fa57c0761a0 2024-03-16T12:20:47.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.557+0000 7fa57aa00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 0x7fa5600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.558+0000 7fa57aa00640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 0x7fa5600a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa57c19f570 tx=0x7fa568005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:47.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.561+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa56c09be60 con 0x7fa57c0761a0 2024-03-16T12:20:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.758+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa57c02d460 con 0x7fa57c0761a0 2024-03-16T12:20:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.760+0000 7fa573e00640 1 -- 172.21.15.98:0/1425032700 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7fa56c09b760 con 0x7fa57c0761a0 2024-03-16T12:20:47.741 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:47.741 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:20:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 msgr2=0x7fa5600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 0x7fa5600a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa57c19f570 tx=0x7fa568005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 msgr2=0x7fa57c19e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e9f0 tx=0x7fa56c00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 shutdown_connections 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa5600a2880 0x7fa5600a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c077d80 0x7fa57c1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa57c0773e0 0x7fa57c19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 --2- 172.21.15.98:0/1425032700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa57c0761a0 0x7fa57c19e060 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 >> 172.21.15.98:0/1425032700 conn(0x7fa57c0fef90 msgr2=0x7fa57c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 shutdown_connections 2024-03-16T12:20:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.763+0000 7fa582600640 1 -- 172.21.15.98:0/1425032700 wait complete. 2024-03-16T12:20:47.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:20:47.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 -- 172.21.15.98:0/2918507906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 msgr2=0x7ff8f0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 --2- 172.21.15.98:0/2918507906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0101c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009900 tx=0x7ff8e002bc30 comp rx=0 tx=0).stop 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 -- 172.21.15.98:0/2918507906 shutdown_connections 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 --2- 172.21.15.98:0/2918507906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 0x7ff8f0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 --2- 172.21.15.98:0/2918507906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 0x7ff8f0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 --2- 172.21.15.98:0/2918507906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0101c10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 -- 172.21.15.98:0/2918507906 >> 172.21.15.98:0/2918507906 conn(0x7ff8f00fcd80 msgr2=0x7ff8f00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 -- 172.21.15.98:0/2918507906 shutdown_connections 2024-03-16T12:20:47.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.979+0000 7ff8f6800640 1 -- 172.21.15.98:0/2918507906 wait complete. 2024-03-16T12:20:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 Processor -- start 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 -- start start 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 0x7ff8f019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 0x7ff8f01a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010d440 con 0x7ff8f01033a0 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010d2c0 con 0x7ff8f01029e0 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.980+0000 7ff8f6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010d5c0 con 0x7ff8f01017f0 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48752/0 (socket says 172.21.15.98:48752) 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 -- 172.21.15.98:0/232464719 learned_addr learned my addr 172.21.15.98:0/232464719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 0x7ff8f019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8efe00640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 0x7ff8f01a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 -- 172.21.15.98:0/232464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 msgr2=0x7ff8f019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 0x7ff8f019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 -- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 msgr2=0x7ff8f01a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 0x7ff8f01a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 -- 172.21.15.98:0/232464719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f01a1950 con 0x7ff8f01017f0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.981+0000 7ff8ef400640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000c00 tx=0x7ff8e002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.982+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0041070 con 0x7ff8f01017f0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.982+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f01a1be0 con 0x7ff8f01017f0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.982+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e0004450 con 0x7ff8f01017f0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.982+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0034440 con 0x7ff8f01017f0 2024-03-16T12:20:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.982+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f01a2120 con 0x7ff8f01017f0 2024-03-16T12:20:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.983+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f01a22f0 con 0x7ff8f01017f0 2024-03-16T12:20:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.986+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff8e0044070 con 0x7ff8f01017f0 2024-03-16T12:20:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.988+0000 7ff8ecc00640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 0x7ff8d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.988+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff8e00f0b70 con 0x7ff8f01017f0 2024-03-16T12:20:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.988+0000 7ff8e7e00640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 0x7ff8d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.988+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8e00bd200 con 0x7ff8f01017f0 2024-03-16T12:20:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:47.989+0000 7ff8e7e00640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 0x7ff8d40a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009bd0 tx=0x7ff8d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.189+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8f01a24c0 con 0x7ff8f01017f0 2024-03-16T12:20:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.189+0000 7ff8ecc00640 1 -- 172.21.15.98:0/232464719 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ff8e00c12a0 con 0x7ff8f01017f0 2024-03-16T12:20:48.170 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:20:48.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 msgr2=0x7ff8d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 0x7ff8d40a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009bd0 tx=0x7ff8d8005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 msgr2=0x7ff8f0199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000c00 tx=0x7ff8e002c640 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 shutdown_connections 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff8d40a2800 0x7ff8d40a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f01033a0 0x7ff8f01a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8f01029e0 0x7ff8f019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 --2- 172.21.15.98:0/232464719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8f01017f0 0x7ff8f0199d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.192+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 >> 172.21.15.98:0/232464719 conn(0x7ff8f00fcd80 msgr2=0x7ff8f0073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.193+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 shutdown_connections 2024-03-16T12:20:48.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.193+0000 7ff8f6800640 1 -- 172.21.15.98:0/232464719 wait complete. 2024-03-16T12:20:48.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:48 smithi098 ceph-mon[31633]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:48.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:48 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/512575343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:48 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1425032700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:20:48.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:48 smithi153 ceph-mon[36901]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:48.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:48 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/512575343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:48 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1425032700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:48 smithi173 ceph-mon[36632]: pgmap v383: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:48.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:48 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/512575343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:48 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1425032700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 -- 172.21.15.98:0/2595201776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 msgr2=0x7efda4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 --2- 172.21.15.98:0/2595201776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4100b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd94009900 tx=0x7efd9402bc40 comp rx=0 tx=0).stop 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 -- 172.21.15.98:0/2595201776 shutdown_connections 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 --2- 172.21.15.98:0/2595201776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 --2- 172.21.15.98:0/2595201776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 --2- 172.21.15.98:0/2595201776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4100b00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 -- 172.21.15.98:0/2595201776 >> 172.21.15.98:0/2595201776 conn(0x7efda40fbc70 msgr2=0x7efda40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 -- 172.21.15.98:0/2595201776 shutdown_connections 2024-03-16T12:20:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.418+0000 7efdac000640 1 -- 172.21.15.98:0/2595201776 wait complete. 2024-03-16T12:20:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 Processor -- start 2024-03-16T12:20:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 -- start start 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efda410d4a0 con 0x7efda41018d0 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efda410d320 con 0x7efda41006e0 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.419+0000 7efdac000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efda410d620 con 0x7efda4102290 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54096/0 (socket says 172.21.15.98:54096) 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda9600640 1 -- 172.21.15.98:0/1726456404 learned_addr learned my addr 172.21.15.98:0/1726456404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 -- 172.21.15.98:0/1726456404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 msgr2=0x7efda4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 -- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 msgr2=0x7efda41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 -- 172.21.15.98:0/1726456404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda419f7a0 con 0x7efda4102290 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda9600640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda8c00640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efdaa000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda419f130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00da10 tx=0x7efd9c00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.420+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00bd70 con 0x7efda4102290 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.421+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9c003e50 con 0x7efda4102290 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.421+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda419fa90 con 0x7efda4102290 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.421+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c0106a0 con 0x7efda4102290 2024-03-16T12:20:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.421+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda419ffe0 con 0x7efda4102290 2024-03-16T12:20:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.422+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda4107ad0 con 0x7efda4102290 2024-03-16T12:20:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.424+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efd9c003970 con 0x7efda4102290 2024-03-16T12:20:48.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.425+0000 7efda2000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 0x7efd880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.425+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efd9c0cf120 con 0x7efda4102290 2024-03-16T12:20:48.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.425+0000 7efda9600640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 0x7efd880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.426+0000 7efda9600640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 0x7efd880a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efd94005ab0 tx=0x7efd9402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:48.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.429+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd9c09be60 con 0x7efda4102290 2024-03-16T12:20:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.622+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efda40623e0 con 0x7efda4102290 2024-03-16T12:20:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.623+0000 7efda2000640 1 -- 172.21.15.98:0/1726456404 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7efd9c09b760 con 0x7efda4102290 2024-03-16T12:20:48.604 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:20:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 msgr2=0x7efd880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 0x7efd880a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efd94005ab0 tx=0x7efd9402cf90 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 msgr2=0x7efda419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda419f130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00da10 tx=0x7efd9c00ded0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 shutdown_connections 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efd880a2880 0x7efd880a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efda4102290 0x7efda419f130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efda41018d0 0x7efda41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 --2- 172.21.15.98:0/1726456404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efda41006e0 0x7efda4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 >> 172.21.15.98:0/1726456404 conn(0x7efda40fbc70 msgr2=0x7efda4104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 shutdown_connections 2024-03-16T12:20:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.626+0000 7efdac000640 1 -- 172.21.15.98:0/1726456404 wait complete. 2024-03-16T12:20:48.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:20:48.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.848+0000 7fefc3400640 1 -- 172.21.15.98:0/1877012307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc1099b0 msgr2=0x7fefbc110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.848+0000 7fefc3400640 1 --2- 172.21.15.98:0/1877012307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc1099b0 0x7fefbc110240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009ca0 tx=0x7fefb402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 -- 172.21.15.98:0/1877012307 shutdown_connections 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 --2- 172.21.15.98:0/1877012307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc1099b0 0x7fefbc110240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 --2- 172.21.15.98:0/1877012307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc108ff0 0x7fefbc109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 --2- 172.21.15.98:0/1877012307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc107e00 0x7fefbc108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 -- 172.21.15.98:0/1877012307 >> 172.21.15.98:0/1877012307 conn(0x7fefbc076da0 msgr2=0x7fefbc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 -- 172.21.15.98:0/1877012307 shutdown_connections 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.849+0000 7fefc3400640 1 -- 172.21.15.98:0/1877012307 wait complete. 2024-03-16T12:20:48.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.850+0000 7fefc3400640 1 Processor -- start 2024-03-16T12:20:48.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.850+0000 7fefc3400640 1 -- start start 2024-03-16T12:20:48.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc108ff0 0x7fefbc19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 0x7fefbc1a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc113a60 con 0x7fefbc107e00 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc1138e0 con 0x7fefbc108ff0 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc3400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc113be0 con 0x7fefbc1099b0 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc0a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefbbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc108ff0 0x7fefbc19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc0a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59584/0 (socket says 172.21.15.98:59584) 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc0a00640 1 -- 172.21.15.98:0/1038452189 learned_addr learned my addr 172.21.15.98:0/1038452189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 0x7fefbc1a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 -- 172.21.15.98:0/1038452189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc108ff0 msgr2=0x7fefbc19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc108ff0 0x7fefbc19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 -- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 msgr2=0x7fefbc19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:48.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc1400640 1 -- 172.21.15.98:0/1038452189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc1a5cc0 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.851+0000 7fefc0a00640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefc1400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 0x7fefbc1a5650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fefb402c630 tx=0x7fefb4032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4041020 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4034cd0 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4039040 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc1a5f50 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.852+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefbc1a6300 con 0x7fefbc1099b0 2024-03-16T12:20:48.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.853+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefbc10f0c0 con 0x7fefbc1099b0 2024-03-16T12:20:48.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.855+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fefb4044020 con 0x7fefbc1099b0 2024-03-16T12:20:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.856+0000 7fefb9600640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 0x7fefa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.856+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefb40f0b50 con 0x7fefbc1099b0 2024-03-16T12:20:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.856+0000 7fefc0a00640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 0x7fefa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.857+0000 7fefc0a00640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 0x7fefa40a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefb00058c0 tx=0x7fefb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:48.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:48.860+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fefb404b020 con 0x7fefbc1099b0 2024-03-16T12:20:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.047+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fefbc02d3d0 con 0x7fefbc1099b0 2024-03-16T12:20:49.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.048+0000 7fefb9600640 1 -- 172.21.15.98:0/1038452189 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fefb40bd8c0 con 0x7fefbc1099b0 2024-03-16T12:20:49.028 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 msgr2=0x7fefa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 0x7fefa40a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefb00058c0 tx=0x7fefb0005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 msgr2=0x7fefbc1a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 0x7fefbc1a5650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fefb402c630 tx=0x7fefb4032f20 comp rx=0 tx=0).stop 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 shutdown_connections 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fefa40a2880 0x7fefa40a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefbc1099b0 0x7fefbc1a5650 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefbc108ff0 0x7fefbc19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 --2- 172.21.15.98:0/1038452189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefbc107e00 0x7fefbc19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 >> 172.21.15.98:0/1038452189 conn(0x7fefbc076da0 msgr2=0x7fefbc10c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 shutdown_connections 2024-03-16T12:20:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.050+0000 7fefc3400640 1 -- 172.21.15.98:0/1038452189 wait complete. 2024-03-16T12:20:49.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:20:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 -- 172.21.15.98:0/108706792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 msgr2=0x7fcb6c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 --2- 172.21.15.98:0/108706792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c109470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009900 tx=0x7fcb5802bc50 comp rx=0 tx=0).stop 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 -- 172.21.15.98:0/108706792 shutdown_connections 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 --2- 172.21.15.98:0/108706792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c1099b0 0x7fcb6c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 --2- 172.21.15.98:0/108706792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c109470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 --2- 172.21.15.98:0/108706792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c107e00 0x7fcb6c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.271+0000 7fcb72c00640 1 -- 172.21.15.98:0/108706792 >> 172.21.15.98:0/108706792 conn(0x7fcb6c076da0 msgr2=0x7fcb6c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.272+0000 7fcb72c00640 1 -- 172.21.15.98:0/108706792 shutdown_connections 2024-03-16T12:20:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.272+0000 7fcb72c00640 1 -- 172.21.15.98:0/108706792 wait complete. 2024-03-16T12:20:49.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.272+0000 7fcb72c00640 1 Processor -- start 2024-03-16T12:20:49.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 -- start start 2024-03-16T12:20:49.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c107e00 0x7fcb6c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c1099b0 0x7fcb6c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c113ad0 con 0x7fcb6c107e00 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c113950 con 0x7fcb6c108ff0 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb72c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c113c50 con 0x7fcb6c1099b0 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c107e00 0x7fcb6c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb6b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb70c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c1099b0 0x7fcb6c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb6b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54130/0 (socket says 172.21.15.98:54130) 2024-03-16T12:20:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.273+0000 7fcb6b400640 1 -- 172.21.15.98:0/3803363789 learned_addr learned my addr 172.21.15.98:0/3803363789 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 -- 172.21.15.98:0/3803363789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c1099b0 msgr2=0x7fcb6c1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c1099b0 0x7fcb6c1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 -- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c107e00 msgr2=0x7fcb6c19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c107e00 0x7fcb6c19e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 -- 172.21.15.98:0/3803363789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c1a5d30 con 0x7fcb6c108ff0 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb6b400640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb5802c260 tx=0x7fcb5802c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:49.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58041070 con 0x7fcb6c108ff0 2024-03-16T12:20:49.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb58004450 con 0x7fcb6c108ff0 2024-03-16T12:20:49.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c1a5fc0 con 0x7fcb6c108ff0 2024-03-16T12:20:49.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58034440 con 0x7fcb6c108ff0 2024-03-16T12:20:49.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.274+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c1a6480 con 0x7fcb6c108ff0 2024-03-16T12:20:49.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.275+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c10f120 con 0x7fcb6c108ff0 2024-03-16T12:20:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.277+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcb58044070 con 0x7fcb6c108ff0 2024-03-16T12:20:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.278+0000 7fcb68c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 0x7fcb500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.278+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcb580f0880 con 0x7fcb6c108ff0 2024-03-16T12:20:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.278+0000 7fcb6be00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 0x7fcb500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.279+0000 7fcb6be00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 0x7fcb500a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c108e50 tx=0x7fcb5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.282+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb580bcf70 con 0x7fcb6c108ff0 2024-03-16T12:20:49.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:49 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/232464719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:49 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1726456404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.467+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcb6c062480 con 0x7fcb6c108ff0 2024-03-16T12:20:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.467+0000 7fcb68c00640 1 -- 172.21.15.98:0/3803363789 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fcb580c1010 con 0x7fcb6c108ff0 2024-03-16T12:20:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 msgr2=0x7fcb500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 0x7fcb500a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c108e50 tx=0x7fcb5c005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 msgr2=0x7fcb6c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb5802c260 tx=0x7fcb5802c670 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 shutdown_connections 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcb500a2800 0x7fcb500a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb6c1099b0 0x7fcb6c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c108ff0 0x7fcb6c19e650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 --2- 172.21.15.98:0/3803363789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb6c107e00 0x7fcb6c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 >> 172.21.15.98:0/3803363789 conn(0x7fcb6c076da0 msgr2=0x7fcb6c10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 shutdown_connections 2024-03-16T12:20:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.470+0000 7fcb72c00640 1 -- 172.21.15.98:0/3803363789 wait complete. 2024-03-16T12:20:49.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:49 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/232464719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:49 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1726456404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:20:49.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:49 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/232464719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:49 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1726456404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 -- 172.21.15.98:0/2600235503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490108ff0 msgr2=0x7f1490109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 --2- 172.21.15.98:0/2600235503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490108ff0 0x7f1490109470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1478009900 tx=0x7f147802bc40 comp rx=0 tx=0).stop 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 -- 172.21.15.98:0/2600235503 shutdown_connections 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 --2- 172.21.15.98:0/2600235503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f1490110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 --2- 172.21.15.98:0/2600235503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490108ff0 0x7f1490109470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 --2- 172.21.15.98:0/2600235503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490107e00 0x7f1490108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 -- 172.21.15.98:0/2600235503 >> 172.21.15.98:0/2600235503 conn(0x7f1490076dc0 msgr2=0x7f14900791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 -- 172.21.15.98:0/2600235503 shutdown_connections 2024-03-16T12:20:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.679+0000 7f1495000640 1 -- 172.21.15.98:0/2600235503 wait complete. 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 Processor -- start 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 -- start start 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490108ff0 0x7f149019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113a80 con 0x7f1490108ff0 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113900 con 0x7f14901099b0 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f1495000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1490113c00 con 0x7f1490107e00 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f148de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.680+0000 7f148e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48822/0 (socket says 172.21.15.98:48822) 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148de00640 1 -- 172.21.15.98:0/2915743415 learned_addr learned my addr 172.21.15.98:0/2915743415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54148/0 (socket says 172.21.15.98:54148) 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f1485400640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490108ff0 0x7f149019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 -- 172.21.15.98:0/2915743415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 msgr2=0x7f149019e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 -- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490108ff0 msgr2=0x7f149019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490108ff0 0x7f149019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 -- 172.21.15.98:0/2915743415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14901a5ce0 con 0x7f14901099b0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148de00640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f148e800640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1488009fd0 tx=0x7f148800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148800ce20 con 0x7f14901099b0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1488005020 con 0x7f14901099b0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14901a5fd0 con 0x7f14901099b0 2024-03-16T12:20:49.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.681+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488010720 con 0x7f14901099b0 2024-03-16T12:20:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.682+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a6490 con 0x7f14901099b0 2024-03-16T12:20:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.682+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f149010f0c0 con 0x7f14901099b0 2024-03-16T12:20:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.684+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1488005540 con 0x7f14901099b0 2024-03-16T12:20:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.685+0000 7f1487400640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 0x7f14740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.686+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14880cf3c0 con 0x7f14901099b0 2024-03-16T12:20:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.686+0000 7f148de00640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 0x7f14740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:49.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.686+0000 7f148de00640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 0x7f14740a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14800104e0 tx=0x7f148000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:49.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.688+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f148809c100 con 0x7f14901099b0 2024-03-16T12:20:49.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.923+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f149002d250 con 0x7f14901099b0 2024-03-16T12:20:49.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.924+0000 7f1487400640 1 -- 172.21.15.98:0/2915743415 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7f148809ba00 con 0x7f14901099b0 2024-03-16T12:20:49.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:49.906 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:20:49.906 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:20:49.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.926+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 msgr2=0x7f14740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.926+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 0x7f14740a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14800104e0 tx=0x7f148000b410 comp rx=0 tx=0).stop 2024-03-16T12:20:49.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.926+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 msgr2=0x7f14901a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:49.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.926+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1488009fd0 tx=0x7f148800ef10 comp rx=0 tx=0).stop 2024-03-16T12:20:49.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 shutdown_connections 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f14740a2800 0x7f14740a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14901099b0 0x7f14901a5670 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1490108ff0 0x7f149019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 --2- 172.21.15.98:0/2915743415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1490107e00 0x7f149019e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 >> 172.21.15.98:0/2915743415 conn(0x7f1490076dc0 msgr2=0x7f149010c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 shutdown_connections 2024-03-16T12:20:49.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:49.927+0000 7f1495000640 1 -- 172.21.15.98:0/2915743415 wait complete. 2024-03-16T12:20:49.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 -- 172.21.15.98:0/686719188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 msgr2=0x7f028c077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 --2- 172.21.15.98:0/686719188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c077840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0278009900 tx=0x7f027802bc40 comp rx=0 tx=0).stop 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 -- 172.21.15.98:0/686719188 shutdown_connections 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 --2- 172.21.15.98:0/686719188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c077d80 0x7f028c078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 --2- 172.21.15.98:0/686719188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c077840 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 --2- 172.21.15.98:0/686719188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c0761a0 0x7f028c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 -- 172.21.15.98:0/686719188 >> 172.21.15.98:0/686719188 conn(0x7f028c0fef90 msgr2=0x7f028c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 -- 172.21.15.98:0/686719188 shutdown_connections 2024-03-16T12:20:50.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.154+0000 7f0291000640 1 -- 172.21.15.98:0/686719188 wait complete. 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.155+0000 7f0291000640 1 Processor -- start 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.155+0000 7f0291000640 1 -- start start 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c077d80 0x7f028c1a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f028c113ab0 con 0x7f028c077d80 2024-03-16T12:20:50.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f028c113930 con 0x7f028c0773e0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0291000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f028c113c30 con 0x7f028c0761a0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54166/0 (socket says 172.21.15.98:54166) 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48846/0 (socket says 172.21.15.98:48846) 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 -- 172.21.15.98:0/2439371415 learned_addr learned my addr 172.21.15.98:0/2439371415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f028a800640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c077d80 0x7f028c1a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 -- 172.21.15.98:0/2439371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 msgr2=0x7f028c19e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 -- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c077d80 msgr2=0x7f028c1a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c077d80 0x7f028c1a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289400640 1 -- 172.21.15.98:0/2439371415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c1a5d70 con 0x7f028c0773e0 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.156+0000 7f0289e00640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:50.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0289400640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f027802c8f0 tx=0x7f027802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278041070 con 0x7f028c0773e0 2024-03-16T12:20:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0278004450 con 0x7f028c0773e0 2024-03-16T12:20:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278003d90 con 0x7f028c0773e0 2024-03-16T12:20:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c1a6000 con 0x7f028c0773e0 2024-03-16T12:20:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.157+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c1a64f0 con 0x7f028c0773e0 2024-03-16T12:20:50.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.158+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028c110390 con 0x7f028c0773e0 2024-03-16T12:20:50.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.160+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0278044070 con 0x7f028c0773e0 2024-03-16T12:20:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.161+0000 7f0282a00640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 0x7f02700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.162+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f02780f0940 con 0x7f028c0773e0 2024-03-16T12:20:50.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.162+0000 7f0289e00640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 0x7f02700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.162+0000 7f0289e00640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 0x7f02700a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f027c009fc0 tx=0x7f027c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:50.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.166+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02780bd620 con 0x7f028c0773e0 2024-03-16T12:20:50.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:50 smithi098 ceph-mon[31633]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:50.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:50 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1038452189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:50 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3803363789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:50 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2915743415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:50.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:50 smithi173 ceph-mon[36632]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:50.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:50 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1038452189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:50 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3803363789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:50 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2915743415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:50.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.365+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f028c0623e0 con 0x7f028c0773e0 2024-03-16T12:20:50.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.366+0000 7f0282a00640 1 -- 172.21.15.98:0/2439371415 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f02780bcf20 con 0x7f028c0773e0 2024-03-16T12:20:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:50.348 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:20:50.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 msgr2=0x7f02700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 0x7f02700a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f027c009fc0 tx=0x7f027c005f90 comp rx=0 tx=0).stop 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 msgr2=0x7f028c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f027802c8f0 tx=0x7f027802c660 comp rx=0 tx=0).stop 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 shutdown_connections 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02700a2880 0x7f02700a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f028c077d80 0x7f028c1a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f028c0773e0 0x7f028c19e690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 --2- 172.21.15.98:0/2439371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f028c0761a0 0x7f028c19e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 >> 172.21.15.98:0/2439371415 conn(0x7f028c0fef90 msgr2=0x7f028c100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 shutdown_connections 2024-03-16T12:20:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.369+0000 7f0291000640 1 -- 172.21.15.98:0/2439371415 wait complete. 2024-03-16T12:20:50.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:20:50.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:50 smithi153 ceph-mon[36901]: pgmap v384: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:50.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:50 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1038452189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:50 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3803363789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:20:50.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:50 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2915743415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.604+0000 7f10d6600640 1 -- 172.21.15.98:0/1000003838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01033a0 msgr2=0x7f10d0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.604+0000 7f10d6600640 1 --2- 172.21.15.98:0/1000003838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01033a0 0x7f10d0109c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ca0 tx=0x7f10c802c100 comp rx=0 tx=0).stop 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 -- 172.21.15.98:0/1000003838 shutdown_connections 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 --2- 172.21.15.98:0/1000003838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01033a0 0x7f10d0109c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 --2- 172.21.15.98:0/1000003838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01029e0 0x7f10d0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 --2- 172.21.15.98:0/1000003838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01017f0 0x7f10d0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 -- 172.21.15.98:0/1000003838 >> 172.21.15.98:0/1000003838 conn(0x7f10d00fcd80 msgr2=0x7f10d00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 -- 172.21.15.98:0/1000003838 shutdown_connections 2024-03-16T12:20:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.605+0000 7f10d6600640 1 -- 172.21.15.98:0/1000003838 wait complete. 2024-03-16T12:20:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 Processor -- start 2024-03-16T12:20:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 -- start start 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01033a0 0x7f10d019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010d450 con 0x7f10d01033a0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010d2d0 con 0x7f10d01029e0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.606+0000 7f10d6600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10d010d5d0 con 0x7f10d01017f0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10cf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10cf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48862/0 (socket says 172.21.15.98:48862) 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54194/0 (socket says 172.21.15.98:54194) 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10cf400640 1 -- 172.21.15.98:0/4121069574 learned_addr learned my addr 172.21.15.98:0/4121069574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10cfe00640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01033a0 0x7f10d019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 -- 172.21.15.98:0/4121069574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 msgr2=0x7f10d0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 -- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01033a0 msgr2=0x7f10d019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01033a0 0x7f10d019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 -- 172.21.15.98:0/4121069574 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d019f680 con 0x7f10d01029e0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10cf400640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.607+0000 7f10c7e00640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10b800d6d0 tx=0x7f10b800db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.608+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8017070 con 0x7f10d01029e0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.608+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10b8009820 con 0x7f10d01029e0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.608+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8004d00 con 0x7f10d01029e0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.608+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d019f970 con 0x7f10d01029e0 2024-03-16T12:20:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.608+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d019fe70 con 0x7f10d01029e0 2024-03-16T12:20:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.609+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d0108ab0 con 0x7f10d01029e0 2024-03-16T12:20:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.611+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f10b80099d0 con 0x7f10d01029e0 2024-03-16T12:20:50.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.612+0000 7f10ccc00640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 0x7f10b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:50.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.612+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f10b80ce8a0 con 0x7f10d01029e0 2024-03-16T12:20:50.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.612+0000 7f10cf400640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 0x7f10b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:50.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.613+0000 7f10cf400640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 0x7f10b40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f10c00104e0 tx=0x7f10c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.616+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10b809af30 con 0x7f10d01029e0 2024-03-16T12:20:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.856+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10d002d1b0 con 0x7f10d01029e0 2024-03-16T12:20:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.857+0000 7f10ccc00640 1 -- 172.21.15.98:0/4121069574 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7f10b809efd0 con 0x7f10d01029e0 2024-03-16T12:20:50.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:20:50.838 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:20:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:20:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 msgr2=0x7f10b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 0x7f10b40a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f10c00104e0 tx=0x7f10c000b410 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 msgr2=0x7f10d0197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10b800d6d0 tx=0x7f10b800db90 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 shutdown_connections 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f10b40a2880 0x7f10b40a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10d01033a0 0x7f10d019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d01029e0 0x7f10d0197fa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 --2- 172.21.15.98:0/4121069574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10d01017f0 0x7f10d0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 >> 172.21.15.98:0/4121069574 conn(0x7f10d00fcd80 msgr2=0x7f10d0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 shutdown_connections 2024-03-16T12:20:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:50.860+0000 7f10d6600640 1 -- 172.21.15.98:0/4121069574 wait complete. 2024-03-16T12:20:50.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T12:18:40.454968+0000', 'modified': '2024-03-16T12:19:07.477601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24599, 'mds_1': 24553, 'mds_2': 24542}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/4013234440', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 4013234440}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 4013234440}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2799668647', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2799668647}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2799668647}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/4083501894', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 4083501894}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 4083501894}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/419588181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 419588181}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 419588181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.173:6839/205594788', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 205594788}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 205594788}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/542416413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 542416413}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 542416413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24553, 24599, 24542]} 2024-03-16T12:20:50.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T12:20:50.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 -- 172.21.15.98:0/2143031303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181029e0 msgr2=0x7fd818102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 --2- 172.21.15.98:0/2143031303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181029e0 0x7fd818102e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd800009860 tx=0x7fd80002bb70 comp rx=0 tx=0).stop 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 -- 172.21.15.98:0/2143031303 shutdown_connections 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 --2- 172.21.15.98:0/2143031303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181033a0 0x7fd818109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 --2- 172.21.15.98:0/2143031303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181029e0 0x7fd818102e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 --2- 172.21.15.98:0/2143031303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 -- 172.21.15.98:0/2143031303 >> 172.21.15.98:0/2143031303 conn(0x7fd8180fcd60 msgr2=0x7fd8180ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 -- 172.21.15.98:0/2143031303 shutdown_connections 2024-03-16T12:20:51.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.096+0000 7fd81d000640 1 -- 172.21.15.98:0/2143031303 wait complete. 2024-03-16T12:20:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.097+0000 7fd81d000640 1 Processor -- start 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.097+0000 7fd81d000640 1 -- start start 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.097+0000 7fd81d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.097+0000 7fd81d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181029e0 0x7fd81819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd81d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181033a0 0x7fd8181a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd81d000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd81810d450 con 0x7fd8181033a0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd81d000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd81810d2d0 con 0x7fd8181029e0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd81d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd81810d5d0 con 0x7fd8181017f0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48878/0 (socket says 172.21.15.98:48878) 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd80d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181029e0 0x7fd81819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 -- 172.21.15.98:0/1955371729 learned_addr learned my addr 172.21.15.98:0/1955371729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd816800640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181033a0 0x7fd8181a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 -- 172.21.15.98:0/1955371729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181029e0 msgr2=0x7fd81819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181029e0 0x7fd81819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 -- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181033a0 msgr2=0x7fd8181a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181033a0 0x7fd8181a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 -- 172.21.15.98:0/1955371729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8181a1940 con 0x7fd8181017f0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.098+0000 7fd815e00640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e9a0 tx=0x7fd80800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.099+0000 7fd80f400640 1 -- 172.21.15.98:0/1955371729 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80800cd80 con 0x7fd8181017f0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.099+0000 7fd80f400640 1 -- 172.21.15.98:0/1955371729 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd808010910 con 0x7fd8181017f0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.099+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8181a1c30 con 0x7fd8181017f0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.099+0000 7fd80f400640 1 -- 172.21.15.98:0/1955371729 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808018a30 con 0x7fd8181017f0 2024-03-16T12:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.099+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8181a2180 con 0x7fd8181017f0 2024-03-16T12:20:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd80f400640 1 -- 172.21.15.98:0/1955371729 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd808010ac0 con 0x7fd8181017f0 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 Processor -- start 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 -- start start 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 0x5572d6b22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b22cc0 0x5572d6b25150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572d6a81c50 con 0x5572d6b1db50 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5572d6a2a9d0 con 0x5572d6b22cc0 2024-03-16T12:20:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.102+0000 7fd81fc80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5572d6a7dcf0 con 0x5572d6b1d160 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48882/0 (socket says 172.21.15.98:48882) 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 -- 172.21.15.98:0/270961899 learned_addr learned my addr 172.21.15.98:0/270961899 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd815e00640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 0x5572d6b22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd80d400640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b22cc0 0x5572d6b25150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 -- 172.21.15.98:0/270961899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b22cc0 msgr2=0x5572d6b25150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b22cc0 0x5572d6b25150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 -- 172.21.15.98:0/270961899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 msgr2=0x5572d6b22650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 0x5572d6b22650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 -- 172.21.15.98:0/270961899 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572d6a44f10 con 0x5572d6b1d160 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd815e00640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 0x5572d6b22650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd80f400640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 0x7fd7fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd816800640 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd810009fd0 tx=0x7fd8100118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f44b6729353ce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.103+0000 7fd807400640 1 -- 172.21.15.98:0/270961899 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810004d20 con 0x5572d6b1d160 2024-03-16T12:20:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd80f400640 1 -- 172.21.15.98:0/1955371729 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8080cf210 con 0x7fd8181017f0 2024-03-16T12:20:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd815e00640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 0x7fd7fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd807400640 1 -- 172.21.15.98:0/270961899 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd810004050 con 0x5572d6b1d160 2024-03-16T12:20:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd807400640 1 -- 172.21.15.98:0/270961899 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810005260 con 0x5572d6b1d160 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 -- 172.21.15.98:0/270961899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 msgr2=0x5572d6b1d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd815e00640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 0x7fd7fc0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd80809b780 tx=0x7fd808000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd810009fd0 tx=0x7fd8100118d0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 -- 172.21.15.98:0/270961899 shutdown_connections 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b22cc0 0x5572d6b25150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1db50 0x5572d6b22650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 --2- 172.21.15.98:0/270961899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1d160 0x5572d6b1d580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.104+0000 7fd81fc80740 1 -- 172.21.15.98:0/270961899 >> 172.21.15.98:0/270961899 conn(0x5572d6b186d0 msgr2=0x5572d6b18ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.105+0000 7fd81fc80740 1 -- 172.21.15.98:0/270961899 shutdown_connections 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.105+0000 7fd81fc80740 1 -- 172.21.15.98:0/270961899 wait complete. 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.105+0000 7fd81fc80740 1 -- start start 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.105+0000 7fd81fc80740 1 Processor -- start 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b27290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b2e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b25c70 con 0x5572d6b22cc0 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5572d6a3de20 con 0x5572d6b1d160 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b25df0 con 0x5572d6b1db50 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd81fc80740 10 client.0 _ll_register_callbacks cb 0x5572d6a3d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48886/0 (socket says 172.21.15.98:48886) 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 -- 172.21.15.98:0/945794384 learned_addr learned my addr 172.21.15.98:0/945794384 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd816800640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b27290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd816800640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b2e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 -- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 msgr2=0x5572d6b27290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b27290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 -- 172.21.15.98:0/945794384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 msgr2=0x5572d6b2e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b2e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 -- 172.21.15.98:0/945794384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572d67962d0 con 0x5572d6b1db50 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd816800640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b2e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd80d400640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd800009860 tx=0x7fd8000097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f9e7a285c52bd5a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.106+0000 7fd7fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800043070 con 0x5572d6b1db50 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd810004050 con 0x5572d6b1db50 2024-03-16T12:20:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd800004040 con 0x5572d6b1db50 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd80004f050 con 0x5572d6b1db50 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd81fc80740 10 client.25162 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5572d68606b0 con 0x5572d6b1db50 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd810004050 con 0x5572d6b1db50 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fd800032760 con 0x5572d6b1db50 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.107+0000 7fd81fc80740 10 client.25162 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd81fc80740 10 client.25162 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd81fc80740 20 client.25162 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b340e0 0x5572d6b35550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd81fc80740 4 client.25162 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572d6adf2b0 con 0x5572d6b340e0 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd816800640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b340e0 0x5572d6b35550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd816800640 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b340e0 0x5572d6b35550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af829cc391fc1c3b server_cookie=a8f854a86b1d6cb8 in_seq=0 out_seq=0 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.108+0000 7fd7fbe00640 10 client.25162 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd7fbe00640 1 -- 172.21.15.98:0/945794384 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572d6adf2b0 con 0x5572d6b340e0 2024-03-16T12:20:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd7fbe00640 10 client.25162 handle_command_reply: tid=0 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 client.25162 shutdown 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 msgr2=0x5572d6b277d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd800009860 tx=0x7fd8000097b0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 shutdown_connections 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b340e0 0x5572d6b35550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b2e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b277d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 --2- 172.21.15.98:0/945794384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b27290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 >> 172.21.15.98:0/945794384 conn(0x5572d6b186d0 msgr2=0x5572d6b17a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.109+0000 7fd7fbe00640 0 client.25162 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.110+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 shutdown_connections 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.110+0000 7fd81fc80740 1 -- 172.21.15.98:0/945794384 wait complete. 2024-03-16T12:20:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.110+0000 7fd81fc80740 20 client.25162 trim_cache size 0 max 16384 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 Processor -- start 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 -- start start 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b171e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b17720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b33890 con 0x5572d6b22cc0 2024-03-16T12:20:51.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b33f60 con 0x5572d6b1d160 2024-03-16T12:20:51.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.129+0000 7fd81fc80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b34630 con 0x5572d6b1db50 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd80d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b171e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd80d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48902/0 (socket says 172.21.15.98:48902) 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd80d400640 1 -- 172.21.15.98:0/1592063497 learned_addr learned my addr 172.21.15.98:0/1592063497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b17720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 -- 172.21.15.98:0/1592063497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 msgr2=0x5572d6b2e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 -- 172.21.15.98:0/1592063497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 msgr2=0x5572d6b17720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b17720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 -- 172.21.15.98:0/1592063497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572d6b351b0 con 0x5572d6b1d160 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd80d400640 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd816800640 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b171e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd810011dd0 tx=0x7fd810009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca7573e2a070245 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.130+0000 7fd806a00640 1 -- 172.21.15.98:0/1592063497 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810009b90 con 0x5572d6b1d160 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd806a00640 1 -- 172.21.15.98:0/1592063497 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81002c130 con 0x5572d6b1d160 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd806a00640 1 -- 172.21.15.98:0/1592063497 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8100042b0 con 0x5572d6b1d160 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 -- 172.21.15.98:0/1592063497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 msgr2=0x5572d6b171e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b171e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd810011dd0 tx=0x7fd810009500 comp rx=0 tx=0).stop 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 -- 172.21.15.98:0/1592063497 shutdown_connections 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b22cc0 0x5572d6b17720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b1db50 0x5572d6b2e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 --2- 172.21.15.98:0/1592063497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1d160 0x5572d6b171e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 -- 172.21.15.98:0/1592063497 >> 172.21.15.98:0/1592063497 conn(0x5572d6b186d0 msgr2=0x5572d6b18ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 -- 172.21.15.98:0/1592063497 shutdown_connections 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.131+0000 7fd81fc80740 1 -- 172.21.15.98:0/1592063497 wait complete. 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 -- start start 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 Processor -- start 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 0x5572d6b2c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 0x5572d6b95e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 0x5572d6b98380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b353d0 con 0x5572d6b1d160 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b34d00 con 0x5572d6b1db50 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5572d6b35aa0 con 0x5572d6b22cc0 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.132+0000 7fd81fc80740 10 client.0 _ll_register_callbacks cb 0x7fd800009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd816800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 0x5572d6b2c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 0x5572d6b95e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd816800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 0x5572d6b2c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59694/0 (socket says 172.21.15.98:59694) 2024-03-16T12:20:51.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd816800640 1 -- 172.21.15.98:0/2826945463 learned_addr learned my addr 172.21.15.98:0/2826945463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd816800640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 0x5572d6b98380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 -- 172.21.15.98:0/2826945463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 msgr2=0x5572d6b98380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 0x5572d6b98380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 -- 172.21.15.98:0/2826945463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 msgr2=0x5572d6b2c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 0x5572d6b2c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 -- 172.21.15.98:0/2826945463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572d6b98a90 con 0x5572d6b1db50 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd816800640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 0x5572d6b98380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd80d400640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 0x5572d6b95e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd80002c0b0 tx=0x7fd8000096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e716e07f853d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.133+0000 7fd7fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800057070 con 0x5572d6b1db50 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd800032840 con 0x5572d6b1db50 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd81002c130 con 0x5572d6b1db50 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd800043050 con 0x5572d6b1db50 2024-03-16T12:20:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 10 client.25178 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5572d6b99d90 con 0x5572d6b1db50 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81002c130 con 0x5572d6b1db50 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fd80002c900 con 0x5572d6b1db50 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 10 client.25178 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 10 client.25178 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 20 client.25178 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b9e920 0x5572d6b9d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 4 client.25178 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.134+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "status"}) -- 0x5572d6a8f190 con 0x5572d6b9e920 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd816800640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b9e920 0x5572d6b9d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd816800640 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b9e920 0x5572d6b9d630 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35a86b37318d1ee2 server_cookie=b3ac5d5f239d5ff3 in_seq=0 out_seq=0 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd7fbe00640 10 client.25178 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd7fbe00640 1 -- 172.21.15.98:0/2826945463 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5572d6a8f190 con 0x5572d6b9e920 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd7fbe00640 10 client.25178 handle_command_reply: tid=0 2024-03-16T12:20:51.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 client.25178 shutdown 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 msgr2=0x5572d6b95e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 0x5572d6b95e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd80002c0b0 tx=0x7fd8000096b0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 shutdown_connections 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5572d6b9e920 0x5572d6b9d630 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5572d6b22cc0 0x5572d6b98380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5572d6b1db50 0x5572d6b95e30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 --2- 172.21.15.98:0/2826945463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572d6b1d160 0x5572d6b2c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd7fbe00640 0 client.25178 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.135+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 >> 172.21.15.98:0/2826945463 conn(0x5572d6b186d0 msgr2=0x5572d6b1b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 shutdown_connections 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81fc80740 1 -- 172.21.15.98:0/2826945463 wait complete. 2024-03-16T12:20:51.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81fc80740 20 client.25178 trim_cache size 0 max 16384 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24542, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 104.21529414699999, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:20:51.122 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 150.81297124 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 msgr2=0x7fd7fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 0x7fd7fc0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd80809b780 tx=0x7fd808000bc0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 msgr2=0x7fd818199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e9a0 tx=0x7fd80800ee60 comp rx=0 tx=0).stop 2024-03-16T12:20:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 shutdown_connections 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd7fc0a2800 0x7fd7fc0a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8181033a0 0x7fd8181a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8181029e0 0x7fd81819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 --2- 172.21.15.98:0/1955371729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8181017f0 0x7fd818199d30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 >> 172.21.15.98:0/1955371729 conn(0x7fd8180fcd60 msgr2=0x7fd8181081a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.136+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 shutdown_connections 2024-03-16T12:20:51.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.137+0000 7fd81d000640 1 -- 172.21.15.98:0/1955371729 wait complete. 2024-03-16T12:20:51.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:51 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2439371415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:51 smithi098 ceph-mon[31633]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:51.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:51 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4121069574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T12:20:51.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:51 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2439371415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:51 smithi173 ceph-mon[36632]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:51.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:51 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4121069574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.358+0000 7ff392000640 1 -- 172.21.15.98:0/2278883641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c103a20 msgr2=0x7ff38c103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.358+0000 7ff392000640 1 --2- 172.21.15.98:0/2278883641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c103a20 0x7ff38c103e40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009ca0 tx=0x7ff37c02c100 comp rx=0 tx=0).stop 2024-03-16T12:20:51.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 -- 172.21.15.98:0/2278883641 shutdown_connections 2024-03-16T12:20:51.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 --2- 172.21.15.98:0/2278883641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1055d0 0x7ff38c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 --2- 172.21.15.98:0/2278883641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c104c10 0x7ff38c105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 --2- 172.21.15.98:0/2278883641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c103a20 0x7ff38c103e40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 -- 172.21.15.98:0/2278883641 >> 172.21.15.98:0/2278883641 conn(0x7ff38c0fef90 msgr2=0x7ff38c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 -- 172.21.15.98:0/2278883641 shutdown_connections 2024-03-16T12:20:51.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.359+0000 7ff392000640 1 -- 172.21.15.98:0/2278883641 wait complete. 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 Processor -- start 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 -- start start 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c103a20 0x7ff38c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1055d0 0x7ff38c1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10f670 con 0x7ff38c103a20 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10f4f0 con 0x7ff38c104c10 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff392000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10f7f0 con 0x7ff38c1055d0 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff382a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff38b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c103a20 0x7ff38c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff382a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54266/0 (socket says 172.21.15.98:54266) 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff382a00640 1 -- 172.21.15.98:0/1006559089 learned_addr learned my addr 172.21.15.98:0/1006559089 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.360+0000 7ff38be00640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1055d0 0x7ff38c1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 -- 172.21.15.98:0/1006559089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1055d0 msgr2=0x7ff38c1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1055d0 0x7ff38c1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 -- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c103a20 msgr2=0x7ff38c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c103a20 0x7ff38c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 -- 172.21.15.98:0/1006559089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c1a1960 con 0x7ff38c104c10 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff382a00640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9f0 tx=0x7ff37400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff388c00640 1 -- 172.21.15.98:0/1006559089 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400cd80 con 0x7ff38c104c10 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff388c00640 1 -- 172.21.15.98:0/1006559089 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374010910 con 0x7ff38c104c10 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c1a1c50 con 0x7ff38c104c10 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff388c00640 1 -- 172.21.15.98:0/1006559089 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374018990 con 0x7ff38c104c10 2024-03-16T12:20:51.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.361+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38c1a21a0 con 0x7ff38c104c10 2024-03-16T12:20:51.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.364+0000 7ff388c00640 1 -- 172.21.15.98:0/1006559089 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff374010ac0 con 0x7ff38c104c10 2024-03-16T12:20:51.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.364+0000 7ff394c66740 1 Processor -- start 2024-03-16T12:20:51.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 -- start start 2024-03-16T12:20:51.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6ceb50 0x5630ac6d3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630ac632c50 con 0x5630ac6ceb50 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630ac5db9d0 con 0x5630ac6ce160 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff394c66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5630ac62ecf0 con 0x5630ac6d3cc0 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff38be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff382a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff38be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54272/0 (socket says 172.21.15.98:54272) 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff38be00640 1 -- 172.21.15.98:0/711327520 learned_addr learned my addr 172.21.15.98:0/711327520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.365+0000 7ff38b400640 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6ceb50 0x5630ac6d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 -- 172.21.15.98:0/711327520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 msgr2=0x5630ac6d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 -- 172.21.15.98:0/711327520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6ceb50 msgr2=0x5630ac6d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6ceb50 0x5630ac6d3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 -- 172.21.15.98:0/711327520 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630ac5f5f10 con 0x5630ac6ce160 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff382a00640 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff388c00640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 0x7ff3700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38be00640 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff38400ac10 tx=0x7ff38400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6985487bb49fdaae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff388c00640 1 -- 172.21.15.98:0/1006559089 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3740cf0b0 con 0x7ff38c104c10 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff38b400640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 0x7ff3700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff37be00640 1 -- 172.21.15.98:0/711327520 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384005220 con 0x5630ac6ce160 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff37be00640 1 -- 172.21.15.98:0/711327520 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384004c80 con 0x5630ac6ce160 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.366+0000 7ff37be00640 1 -- 172.21.15.98:0/711327520 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384018840 con 0x5630ac6ce160 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 -- 172.21.15.98:0/711327520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 msgr2=0x5630ac6ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff38b400640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 0x7ff3700a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff37c02c640 tx=0x7ff37c032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff38400ac10 tx=0x7ff38400f790 comp rx=0 tx=0).stop 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 -- 172.21.15.98:0/711327520 shutdown_connections 2024-03-16T12:20:51.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6ceb50 0x5630ac6d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 --2- 172.21.15.98:0/711327520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6ce580 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 -- 172.21.15.98:0/711327520 >> 172.21.15.98:0/711327520 conn(0x5630ac6c96d0 msgr2=0x5630ac6c9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 -- 172.21.15.98:0/711327520 shutdown_connections 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.367+0000 7ff394c66740 1 -- 172.21.15.98:0/711327520 wait complete. 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 -- start start 2024-03-16T12:20:51.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 Processor -- start 2024-03-16T12:20:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6d87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6df810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6d6c70 con 0x5630ac6d3cc0 2024-03-16T12:20:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630ac5eee20 con 0x5630ac6ce160 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.368+0000 7ff394c66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6d6df0 con 0x5630ac6ceb50 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff394c66740 10 client.0 _ll_register_callbacks cb 0x5630ac5ee7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff38b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6d87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff38b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54274/0 (socket says 172.21.15.98:54274) 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff38b400640 1 -- 172.21.15.98:0/2280226847 learned_addr learned my addr 172.21.15.98:0/2280226847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff38be00640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 -- 172.21.15.98:0/2280226847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 msgr2=0x5630ac6d8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 -- 172.21.15.98:0/2280226847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 msgr2=0x5630ac6df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6df810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 -- 172.21.15.98:0/2280226847 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630ac3472d0 con 0x5630ac6ceb50 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff38b400640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff382a00640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6d87d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c6a0 tx=0x7ff374097aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f8c9b3c921785d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff378c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37409b0a0 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.369+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3740d2c60 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff384004c80 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff3740d2e60 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 10 client.25180 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5630ac4116b0 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384004c80 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff37409ab90 con 0x5630ac6ceb50 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 10 client.25180 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 10 client.25180 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 20 client.25180 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac6e4fc0 0x5630ac6e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 4 client.25180 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.370+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630ac6902b0 con 0x5630ac6e4fc0 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.371+0000 7ff38b400640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac6e4fc0 0x5630ac6e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.371+0000 7ff38b400640 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac6e4fc0 0x5630ac6e6430 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af5764ee67ebf97d server_cookie=894175272be12a9b in_seq=0 out_seq=0 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.371+0000 7ff378c00640 10 client.25180 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff378c00640 1 -- 172.21.15.98:0/2280226847 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630ac6902b0 con 0x5630ac6e4fc0 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff378c00640 10 client.25180 handle_command_reply: tid=0 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 client.25180 shutdown 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 msgr2=0x5630ac6d87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6d87d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c6a0 tx=0x7ff374097aa0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 shutdown_connections 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac6e4fc0 0x5630ac6e6430 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6d87d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 --2- 172.21.15.98:0/2280226847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 >> 172.21.15.98:0/2280226847 conn(0x5630ac6c96d0 msgr2=0x5630ac6c8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff378c00640 0 client.25180 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 shutdown_connections 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 1 -- 172.21.15.98:0/2280226847 wait complete. 2024-03-16T12:20:51.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.372+0000 7ff394c66740 20 client.25180 trim_cache size 0 max 16384 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.391+0000 7ff394c66740 1 Processor -- start 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 -- start start 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e4770 con 0x5630ac6d3cc0 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e4e40 con 0x5630ac6ce160 2024-03-16T12:20:51.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff394c66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e5510 con 0x5630ac6ceb50 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff38b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff38b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54286/0 (socket says 172.21.15.98:54286) 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff38b400640 1 -- 172.21.15.98:0/2300212485 learned_addr learned my addr 172.21.15.98:0/2300212485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff382a00640 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.392+0000 7ff38be00640 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 -- 172.21.15.98:0/2300212485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 msgr2=0x5630ac6d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 -- 172.21.15.98:0/2300212485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 msgr2=0x5630ac6d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 -- 172.21.15.98:0/2300212485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630ac69f750 con 0x5630ac6ceb50 2024-03-16T12:20:51.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff382a00640 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6df7d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9c0 tx=0x7ff374097b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f9c4c9b3ea0fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff37b400640 1 -- 172.21.15.98:0/2300212485 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740a3e30 con 0x5630ac6ceb50 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff37b400640 1 -- 172.21.15.98:0/2300212485 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37409f7e0 con 0x5630ac6ceb50 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.393+0000 7ff37b400640 1 -- 172.21.15.98:0/2300212485 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740d2180 con 0x5630ac6ceb50 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- 172.21.15.98:0/2300212485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 msgr2=0x5630ac6df7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6df7d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9c0 tx=0x7ff374097b10 comp rx=0 tx=0).stop 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- 172.21.15.98:0/2300212485 shutdown_connections 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6d78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ceb50 0x5630ac6df7d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 --2- 172.21.15.98:0/2300212485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ce160 0x5630ac6d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- 172.21.15.98:0/2300212485 >> 172.21.15.98:0/2300212485 conn(0x5630ac6c96d0 msgr2=0x5630ac6c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- 172.21.15.98:0/2300212485 shutdown_connections 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- 172.21.15.98:0/2300212485 wait complete. 2024-03-16T12:20:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 -- start start 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.394+0000 7ff394c66740 1 Processor -- start 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e62b0 con 0x5630ac6d3cc0 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e5be0 con 0x5630ac6ceb50 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5630ac6e6980 con 0x5630ac6ce160 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff394c66740 10 client.0 _ll_register_callbacks cb 0x7ff374009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff38be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54292/0 (socket says 172.21.15.98:54292) 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48968/0 (socket says 172.21.15.98:48968) 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 -- 172.21.15.98:0/1671169993 learned_addr learned my addr 172.21.15.98:0/1671169993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 -- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 msgr2=0x5630ac6dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 -- 172.21.15.98:0/1671169993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 msgr2=0x5630ac6de160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6de160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.395+0000 7ff382a00640 1 -- 172.21.15.98:0/1671169993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630ac6de750 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff38b400640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff382a00640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff37409af70 tx=0x7ff3740978f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe03cfb7e06909e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff378c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740a3e30 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff374018b30 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff37409f7e0 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff374018d70 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff394c66740 10 client.25196 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5630ac6dfa50 con 0x5630ac6ceb50 2024-03-16T12:20:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.396+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37409f7e0 con 0x5630ac6ceb50 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff3740a0700 con 0x5630ac6ceb50 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 10 client.25196 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 10 client.25196 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 20 client.25196 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac74f020 0x5630ac74bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 4 client.25196 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "status"}) -- 0x5630ac640190 con 0x5630ac74f020 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff38b400640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac74f020 0x5630ac74bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff38b400640 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac74f020 0x5630ac74bb30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c4635ebb9e4cfe1 server_cookie=7e4177267ee6ee61 in_seq=0 out_seq=0 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff378c00640 10 client.25196 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:20:51.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff378c00640 1 -- 172.21.15.98:0/1671169993 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5630ac640190 con 0x5630ac74f020 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.397+0000 7ff378c00640 10 client.25196 handle_command_reply: tid=0 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 client.25196 shutdown 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 msgr2=0x5630ac6db140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff37409af70 tx=0x7ff3740978f0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 shutdown_connections 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5630ac74f020 0x5630ac74bb30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630ac6d3cc0 0x5630ac6de160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5630ac6ceb50 0x5630ac6db140 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff378c00640 0 client.25196 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 --2- 172.21.15.98:0/1671169993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5630ac6ce160 0x5630ac6dac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 >> 172.21.15.98:0/1671169993 conn(0x5630ac6c96d0 msgr2=0x5630ac6d78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 shutdown_connections 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 1 -- 172.21.15.98:0/1671169993 wait complete. 2024-03-16T12:20:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.398+0000 7ff394c66740 20 client.25196 trim_cache size 0 max 16384 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24553, 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 105.50060799400001, 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:20:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:20:51.386 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148.199487952 2024-03-16T12:20:51.386 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:20:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 msgr2=0x7ff3700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 0x7ff3700a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff37c02c640 tx=0x7ff37c032340 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 msgr2=0x7ff38c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9f0 tx=0x7ff37400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 shutdown_connections 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2880 0x7ff3700a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1055d0 0x7ff38c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c104c10 0x7ff38c19a280 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 --2- 172.21.15.98:0/1006559089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c103a20 0x7ff38c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 >> 172.21.15.98:0/1006559089 conn(0x7ff38c0fef90 msgr2=0x7ff38c107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 shutdown_connections 2024-03-16T12:20:51.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.399+0000 7ff392000640 1 -- 172.21.15.98:0/1006559089 wait complete. 2024-03-16T12:20:51.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T12:20:51.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:51 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2439371415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:51 smithi153 ceph-mon[36901]: pgmap v385: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:20:51.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:51 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4121069574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 -- 172.21.15.98:0/4057052606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8041055d0 msgr2=0x7ff80410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 --2- 172.21.15.98:0/4057052606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8041055d0 0x7ff80410be60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009ca0 tx=0x7ff7fc02c100 comp rx=0 tx=0).stop 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 -- 172.21.15.98:0/4057052606 shutdown_connections 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 --2- 172.21.15.98:0/4057052606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8041055d0 0x7ff80410be60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 --2- 172.21.15.98:0/4057052606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 0x7ff804105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 --2- 172.21.15.98:0/4057052606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff804103a20 0x7ff804103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 -- 172.21.15.98:0/4057052606 >> 172.21.15.98:0/4057052606 conn(0x7ff8040fef90 msgr2=0x7ff8041013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 -- 172.21.15.98:0/4057052606 shutdown_connections 2024-03-16T12:20:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.619+0000 7ff809e00640 1 -- 172.21.15.98:0/4057052606 wait complete. 2024-03-16T12:20:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 Processor -- start 2024-03-16T12:20:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 -- start start 2024-03-16T12:20:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 0x7ff80419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8041055d0 0x7ff8041a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff80410f6a0 con 0x7ff804104c10 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff80410f520 con 0x7ff8041055d0 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff809e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff80410f820 con 0x7ff804103a20 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff802000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 0x7ff80419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff802a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff803400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8041055d0 0x7ff8041a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff802a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48982/0 (socket says 172.21.15.98:48982) 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.620+0000 7ff802a00640 1 -- 172.21.15.98:0/1017878523 learned_addr learned my addr 172.21.15.98:0/1017878523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 -- 172.21.15.98:0/1017878523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8041055d0 msgr2=0x7ff8041a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8041055d0 0x7ff8041a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 -- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 msgr2=0x7ff80419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 0x7ff80419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 -- 172.21.15.98:0/1017878523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8041a1940 con 0x7ff804103a20 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff802a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e9c0 tx=0x7ff7f400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff808a00640 1 -- 172.21.15.98:0/1017878523 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f400cd00 con 0x7ff804103a20 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff808a00640 1 -- 172.21.15.98:0/1017878523 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f40035f0 con 0x7ff804103a20 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff808a00640 1 -- 172.21.15.98:0/1017878523 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4010690 con 0x7ff804103a20 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.621+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8041a1c30 con 0x7ff804103a20 2024-03-16T12:20:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.622+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8041a2130 con 0x7ff804103a20 2024-03-16T12:20:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 Processor -- start 2024-03-16T12:20:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff808a00640 1 -- 172.21.15.98:0/1017878523 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff7f40037a0 con 0x7ff804103a20 2024-03-16T12:20:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 -- start start 2024-03-16T12:20:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf95650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 0x55e0baf98150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0baef4c50 con 0x55e0baf90b50 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0bae9d9d0 con 0x55e0baf95cc0 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.626+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0baef0cf0 con 0x55e0baf90160 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff802a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf95650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48998/0 (socket says 172.21.15.98:48998) 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 -- 172.21.15.98:0/774663799 learned_addr learned my addr 172.21.15.98:0/774663799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff802000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 0x55e0baf98150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 -- 172.21.15.98:0/774663799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 msgr2=0x55e0baf98150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 0x55e0baf98150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 -- 172.21.15.98:0/774663799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 msgr2=0x55e0baf95650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf95650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 -- 172.21.15.98:0/774663799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0baeb7f10 con 0x55e0baf90160 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff802000640 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 0x55e0baf98150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.627+0000 7ff803400640 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc005e30 tx=0x7ff7fc009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64023cc70dc1155a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff7efe00640 1 -- 172.21.15.98:0/774663799 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc043070 con 0x55e0baf90160 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff808a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 0x7ff7e80a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff7efe00640 1 -- 172.21.15.98:0/774663799 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc003970 con 0x55e0baf90160 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff7efe00640 1 -- 172.21.15.98:0/774663799 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc039040 con 0x55e0baf90160 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff808a00640 1 -- 172.21.15.98:0/1017878523 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7f40cf0d0 con 0x7ff804103a20 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.628+0000 7ff802a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 0x7ff7e80a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- 172.21.15.98:0/774663799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 msgr2=0x55e0baf90580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc005e30 tx=0x7ff7fc009800 comp rx=0 tx=0).stop 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff802a00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 0x7ff7e80a4d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40ccbe0 tx=0x7ff7f40d2bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- 172.21.15.98:0/774663799 shutdown_connections 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf95cc0 0x55e0baf98150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf95650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/774663799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90160 0x55e0baf90580 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- 172.21.15.98:0/774663799 >> 172.21.15.98:0/774663799 conn(0x55e0baf8b6d0 msgr2=0x55e0baf8bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- 172.21.15.98:0/774663799 shutdown_connections 2024-03-16T12:20:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- 172.21.15.98:0/774663799 wait complete. 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 -- start start 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.629+0000 7ff80cc0c740 1 Processor -- start 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bafa17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0baf98c10 con 0x55e0baf90b50 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0baeb0e20 con 0x55e0baf90160 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0baf98d90 con 0x55e0baf95cc0 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff80cc0c740 10 client.0 _ll_register_callbacks cb 0x55e0baeb07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff803400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59802/0 (socket says 172.21.15.98:59802) 2024-03-16T12:20:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 -- 172.21.15.98:0/3989485501 learned_addr learned my addr 172.21.15.98:0/3989485501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff803400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54328/0 (socket says 172.21.15.98:54328) 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff803400640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bafa17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 -- 172.21.15.98:0/3989485501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 msgr2=0x55e0bafa17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bafa17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.630+0000 7ff802000640 1 -- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 msgr2=0x55e0baf9a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff802000640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff802000640 1 -- 172.21.15.98:0/3989485501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bac092d0 con 0x55e0baf90b50 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff803400640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff802000640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0012740 tx=0x7ff7f0012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992f59d782ae380 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff7ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0004030 con 0x55e0baf90b50 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7f00041d0 con 0x55e0baf90b50 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7fc003970 con 0x55e0baf90b50 2024-03-16T12:20:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff7f0025050 con 0x55e0baf90b50 2024-03-16T12:20:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff80cc0c740 10 client.15081 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.631+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e0bacd36b0 con 0x55e0baf90b50 2024-03-16T12:20:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc003970 con 0x55e0baf90b50 2024-03-16T12:20:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff7f0003ac0 con 0x55e0baf90b50 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 10 client.15081 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 20 client.15081 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bafa7130 0x55e0bafa85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 4 client.15081 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0baf522b0 con 0x55e0bafa7130 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.632+0000 7ff803400640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bafa7130 0x55e0bafa85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.633+0000 7ff803400640 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bafa7130 0x55e0bafa85a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf2e93b3593adbb server_cookie=4f0cef9278585b2 in_seq=0 out_seq=0 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.633+0000 7ff7ecc00640 10 client.15081 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff7ecc00640 1 -- 172.21.15.98:0/3989485501 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e0baf522b0 con 0x55e0bafa7130 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff7ecc00640 10 client.15081 handle_command_reply: tid=0 2024-03-16T12:20:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 client.15081 shutdown 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 msgr2=0x55e0baf9a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0012740 tx=0x7ff7f0012c00 comp rx=0 tx=0).stop 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 shutdown_connections 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bafa7130 0x55e0bafa85a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bafa17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90b50 0x55e0baf9a850 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/3989485501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf9a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff7ecc00640 0 client.15081 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 >> 172.21.15.98:0/3989485501 conn(0x55e0baf8b6d0 msgr2=0x55e0baf8aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 shutdown_connections 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 1 -- 172.21.15.98:0/3989485501 wait complete. 2024-03-16T12:20:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.634+0000 7ff80cc0c740 20 client.15081 trim_cache size 0 max 16384 2024-03-16T12:20:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 Processor -- start 2024-03-16T12:20:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 -- start start 2024-03-16T12:20:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf99930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf95cc0 0x55e0baf9a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa68e0 con 0x55e0baf95cc0 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa6fb0 con 0x55e0baf90160 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.654+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa7680 con 0x55e0baf90b50 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49012/0 (socket says 172.21.15.98:49012) 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 -- 172.21.15.98:0/2236472634 learned_addr learned my addr 172.21.15.98:0/2236472634 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff803400640 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf99930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff803400640 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf95cc0 0x55e0baf9a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 -- 172.21.15.98:0/2236472634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 msgr2=0x55e0baf99930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf99930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 -- 172.21.15.98:0/2236472634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf95cc0 msgr2=0x55e0baf9a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf95cc0 0x55e0baf9a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 -- 172.21.15.98:0/2236472634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0baf61750 con 0x55e0baf90b50 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff802000640 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0003f60 tx=0x7ff7f00107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebc518640277fce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.655+0000 7ff7ef400640 1 -- 172.21.15.98:0/2236472634 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0022040 con 0x55e0baf90b50 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff7ef400640 1 -- 172.21.15.98:0/2236472634 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f0012e50 con 0x55e0baf90b50 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff7ef400640 1 -- 172.21.15.98:0/2236472634 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0005370 con 0x55e0baf90b50 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 -- 172.21.15.98:0/2236472634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 msgr2=0x55e0bafa1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0003f60 tx=0x7ff7f00107e0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 -- 172.21.15.98:0/2236472634 shutdown_connections 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf95cc0 0x55e0baf9a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf90b50 0x55e0bafa1770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/2236472634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90160 0x55e0baf99930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 -- 172.21.15.98:0/2236472634 >> 172.21.15.98:0/2236472634 conn(0x55e0baf8b6d0 msgr2=0x55e0baf8bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 -- 172.21.15.98:0/2236472634 shutdown_connections 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.656+0000 7ff80cc0c740 1 -- 172.21.15.98:0/2236472634 wait complete. 2024-03-16T12:20:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 -- start start 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 Processor -- start 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 0x55e0baf9f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bb00b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa8420 con 0x55e0baf90160 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa7d50 con 0x55e0baf90b50 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.657+0000 7ff80cc0c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0bafa8af0 con 0x55e0baf95cc0 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff80cc0c740 10 client.0 _ll_register_callbacks cb 0x7ff7f0009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff803400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 0x55e0baf9f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54354/0 (socket says 172.21.15.98:54354) 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff803400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 0x55e0baf9f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59808/0 (socket says 172.21.15.98:59808) 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 -- 172.21.15.98:0/1940590420 learned_addr learned my addr 172.21.15.98:0/1940590420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff803400640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bb00b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 -- 172.21.15.98:0/1940590420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 msgr2=0x55e0bb00b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bb00b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 -- 172.21.15.98:0/1940590420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 msgr2=0x55e0baf9f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 0x55e0baf9f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 -- 172.21.15.98:0/1940590420 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bb00bb80 con 0x55e0baf90b50 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff802000640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0012490 tx=0x7ff7f0010b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891f56de2c850e5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.658+0000 7ff7ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0011ba0 con 0x55e0baf90b50 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7f0011d40 con 0x55e0baf90b50 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7f0012e50 con 0x55e0baf90b50 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff7f0005110 con 0x55e0baf90b50 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 10 client.25214 fetch_fsmap learned FSMap version 23 2024-03-16T12:20:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e0bb00ce80 con 0x55e0baf90b50 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f0012e50 con 0x55e0baf90b50 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff7f0031030 con 0x55e0baf90b50 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 10 client.25214 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 10 client.25214 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 20 client.25214 populate_metadata read hostname 'smithi098' 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bb0116c0 0x55e0bb0103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 4 client.25214 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.659+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "status"}) -- 0x55e0baf02190 con 0x55e0bb0116c0 2024-03-16T12:20:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff803400640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bb0116c0 0x55e0bb0103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff803400640 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bb0116c0 0x55e0bb0103d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=137a9baedf1be338 server_cookie=434545aa64ad6cbd in_seq=0 out_seq=0 2024-03-16T12:20:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff7ecc00640 10 client.25214 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff7ecc00640 1 -- 172.21.15.98:0/1940590420 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e0baf02190 con 0x55e0bb0116c0 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff7ecc00640 10 client.25214 handle_command_reply: tid=0 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff80cc0c740 1 client.25214 shutdown 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 msgr2=0x55e0bb008e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.660+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0012490 tx=0x7ff7f0010b00 comp rx=0 tx=0).stop 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 shutdown_connections 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e0bb0116c0 0x55e0bb0103d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0baf95cc0 0x55e0bb00b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e0baf90b50 0x55e0bb008e80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 --2- 172.21.15.98:0/1940590420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0baf90160 0x55e0baf9f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff7ecc00640 0 client.25214 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:20:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 >> 172.21.15.98:0/1940590420 conn(0x55e0baf8b6d0 msgr2=0x55e0baf99f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 shutdown_connections 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 1 -- 172.21.15.98:0/1940590420 wait complete. 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff80cc0c740 20 client.25214 trim_cache size 0 max 16384 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24599, 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:20:51.647 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 130.63506106400001, 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 137.308078779 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:20:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 msgr2=0x7ff7e80a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 0x7ff7e80a4d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40ccbe0 tx=0x7ff7f40d2bb0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 msgr2=0x7ff804199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.661+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400e9c0 tx=0x7ff7f400ee80 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 shutdown_connections 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7e80a28d0 0x7ff7e80a4d90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8041055d0 0x7ff8041a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff804104c10 0x7ff80419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 --2- 172.21.15.98:0/1017878523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff804103a20 0x7ff804199d30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 >> 172.21.15.98:0/1017878523 conn(0x7ff8040fef90 msgr2=0x7ff804107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 shutdown_connections 2024-03-16T12:20:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:51.662+0000 7ff809e00640 1 -- 172.21.15.98:0/1017878523 wait complete. 2024-03-16T12:20:51.659 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T12:20:51.659 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T12:20:51.673 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:20:51.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T12:20:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:54.135 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:53 smithi098 ceph-mon[31633]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:54.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:53 smithi153 ceph-mon[36901]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:54.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:53 smithi173 ceph-mon[36632]: pgmap v386: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.475+0000 7f5e5d000700 1 -- 172.21.15.98:0/1854579470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103eb0 msgr2=0x7f5e5810a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.475+0000 7f5e5d000700 1 --2- 172.21.15.98:0/1854579470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103eb0 0x7f5e5810a760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009990 tx=0x7f5e50009ca0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 -- 172.21.15.98:0/1854579470 shutdown_connections 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 --2- 172.21.15.98:0/1854579470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58102310 0x7f5e58102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 --2- 172.21.15.98:0/1854579470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103eb0 0x7f5e5810a760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 --2- 172.21.15.98:0/1854579470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58103510 0x7f5e58103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 -- 172.21.15.98:0/1854579470 >> 172.21.15.98:0/1854579470 conn(0x7f5e580fd870 msgr2=0x7f5e580ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 -- 172.21.15.98:0/1854579470 shutdown_connections 2024-03-16T12:20:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.476+0000 7f5e5d000700 1 -- 172.21.15.98:0/1854579470 wait complete. 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.477+0000 7f5e5d000700 1 Processor -- start 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.477+0000 7f5e5d000700 1 -- start start 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.477+0000 7f5e5d000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e5d000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 0x7f5e5819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e5d000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 0x7f5e581a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e5d000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5810df30 con 0x7f5e58102310 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e5d000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5810ddb0 con 0x7f5e58103eb0 2024-03-16T12:20:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e5d000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5810e0b0 con 0x7f5e58103510 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e55e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 0x7f5e5819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e55e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59834/0 (socket says 172.21.15.98:59834) 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e55e00700 1 -- 172.21.15.98:0/3648205335 learned_addr learned my addr 172.21.15.98:0/3648205335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e56800700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 0x7f5e581a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 -- 172.21.15.98:0/3648205335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 msgr2=0x7f5e581a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 0x7f5e581a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 -- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 msgr2=0x7f5e5819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e4d400700 1 -- 172.21.15.98:0/3648205335 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e581a2330 con 0x7f5e58103510 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e55e00700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.478+0000 7f5e56800700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 0x7f5e581a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:20:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e4d400700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 0x7f5e5819ac10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000ea20 tx=0x7f5e4000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:54.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000cbd0 con 0x7f5e58103510 2024-03-16T12:20:54.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4000ee60 con 0x7f5e58103510 2024-03-16T12:20:54.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40018680 con 0x7f5e58103510 2024-03-16T12:20:54.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e581a2620 con 0x7f5e58103510 2024-03-16T12:20:54.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.479+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e581a2b10 con 0x7f5e58103510 2024-03-16T12:20:54.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.480+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e58068f90 con 0x7f5e58103510 2024-03-16T12:20:54.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.482+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5e40018820 con 0x7f5e58103510 2024-03-16T12:20:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.483+0000 7f5e4f400700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 0x7f5e3c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.483+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e400cee80 con 0x7f5e58103510 2024-03-16T12:20:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.484+0000 7f5e55e00700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 0x7f5e3c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.484+0000 7f5e55e00700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 0x7f5e3c0a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e9c0 tx=0x7f5e4800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.487+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e4009bb60 con 0x7f5e58103510 2024-03-16T12:20:54.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.726+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f5e58075c20 con 0x7f5e3c0a2910 2024-03-16T12:20:54.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.728+0000 7f5e4f400700 1 -- 172.21.15.98:0/3648205335 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5e58075c20 con 0x7f5e3c0a2910 2024-03-16T12:20:54.710 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-03-16T12:20:54.710 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-03-16T12:20:54.710 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 msgr2=0x7f5e3c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 0x7f5e3c0a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800e9c0 tx=0x7f5e4800b040 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 msgr2=0x7f5e5819ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 0x7f5e5819ac10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000ea20 tx=0x7f5e4000ed30 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 shutdown_connections 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5e3c0a2910 0x7f5e3c0a4dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e58103eb0 0x7f5e581a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e58102310 0x7f5e5819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 --2- 172.21.15.98:0/3648205335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e58103510 0x7f5e5819ac10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.732+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 >> 172.21.15.98:0/3648205335 conn(0x7f5e580fd870 msgr2=0x7f5e58074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.733+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 shutdown_connections 2024-03-16T12:20:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:54.733+0000 7f5e5d000700 1 -- 172.21.15.98:0/3648205335 wait complete. 2024-03-16T12:20:55.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T12:20:55.970 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:20:56.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:55 smithi098 ceph-mon[31633]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:20:56.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:55 smithi098 ceph-mon[31633]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:56.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:55 smithi153 ceph-mon[36901]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:20:56.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:55 smithi153 ceph-mon[36901]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:56.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:55 smithi173 ceph-mon[36632]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:20:56.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:55 smithi173 ceph-mon[36632]: pgmap v387: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:20:57.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:57 smithi098 ceph-mon[31633]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:58.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 -- 172.21.15.98:0/2866209057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 msgr2=0x7fe38c074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 --2- 172.21.15.98:0/2866209057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c074760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00b0a0 tx=0x7fe37c00b3b0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 -- 172.21.15.98:0/2866209057 shutdown_connections 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 --2- 172.21.15.98:0/2866209057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c074760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 --2- 172.21.15.98:0/2866209057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c10e8f0 0x7fe38c110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 --2- 172.21.15.98:0/2866209057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c074d30 0x7fe38c0751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.238+0000 7fe393e00700 1 -- 172.21.15.98:0/2866209057 >> 172.21.15.98:0/2866209057 conn(0x7fe38c0fd870 msgr2=0x7fe38c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.239+0000 7fe393e00700 1 -- 172.21.15.98:0/2866209057 shutdown_connections 2024-03-16T12:20:58.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.239+0000 7fe393e00700 1 -- 172.21.15.98:0/2866209057 wait complete. 2024-03-16T12:20:58.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.240+0000 7fe393e00700 1 Processor -- start 2024-03-16T12:20:58.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.240+0000 7fe393e00700 1 -- start start 2024-03-16T12:20:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 0x7fe38c1a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c114540 con 0x7fe38c10e8f0 2024-03-16T12:20:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c1143c0 con 0x7fe38c076300 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe393e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c1146c0 con 0x7fe38c074d30 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49500/0 (socket says 172.21.15.98:49500) 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 -- 172.21.15.98:0/3562090233 learned_addr learned my addr 172.21.15.98:0/3562090233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe390a00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 0x7fe38c1a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 -- 172.21.15.98:0/3562090233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 msgr2=0x7fe38c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 -- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 msgr2=0x7fe38c1a6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 0x7fe38c1a6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391400700 1 -- 172.21.15.98:0/3562090233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38c1a67e0 con 0x7fe38c074d30 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.241+0000 7fe391e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 0x7fe38c1a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:20:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe391400700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00b070 tx=0x7fe37c004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c021070 con 0x7fe38c074d30 2024-03-16T12:20:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37c007b80 con 0x7fe38c074d30 2024-03-16T12:20:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38c1a6a70 con 0x7fe38c074d30 2024-03-16T12:20:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c01d450 con 0x7fe38c074d30 2024-03-16T12:20:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.242+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe38c1a6f20 con 0x7fe38c074d30 2024-03-16T12:20:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.243+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe38c068f90 con 0x7fe38c074d30 2024-03-16T12:20:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.245+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe37c004020 con 0x7fe38c074d30 2024-03-16T12:20:58.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.247+0000 7fe38a000700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 0x7fe3700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:20:58.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.247+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe37c00f020 con 0x7fe38c074d30 2024-03-16T12:20:58.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.247+0000 7fe390a00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 0x7fe3700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:20:58.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.248+0000 7fe390a00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 0x7fe3700a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe38c076030 tx=0x7fe378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:20:58.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.251+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe37c02a020 con 0x7fe38c074d30 2024-03-16T12:20:58.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:57 smithi153 ceph-mon[36901]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:58.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:57 smithi173 ceph-mon[36632]: pgmap v388: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:20:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.482+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe38c062710 con 0x7fe3700a2890 2024-03-16T12:20:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.544+0000 7fe38a000700 1 -- 172.21.15.98:0/3562090233 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe38c062710 con 0x7fe3700a2890 2024-03-16T12:20:58.526 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (2m) 39s ago 2m 12.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ef6b0087c4e3 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi153 running (2m) 75s ago 2m 26.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7767e6eb6257 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi173 running (2m) 75s ago 2m 16.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b edfdbe6d26ed 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (2m) 39s ago 2m 12.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a54ee58bae3d 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi153 running (2m) 75s ago 2m 16.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 77f5946986a9 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi173 running (2m) 75s ago 2m 15.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 31d8c66a952f 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (2m) 39s ago 2m 12.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7b3568f197c2 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi153 running (2m) 75s ago 2m 18.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b29b1ba93086 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi173 running (2m) 75s ago 2m 18.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 79d40dcf9895 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (2m) 39s ago 2m 12.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b245bc00bcc2 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi153 running (2m) 75s ago 2m 13.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 634be9d16f5d 2024-03-16T12:20:58.527 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi173 running (3m) 75s ago 3m 15.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 01869771ba2d 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (11m) 39s ago 11m 515M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 1661ffdd4fff 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi153 *:8443,8765 running (8m) 75s ago 8m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 87b50ce54ad1 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi173 *:8443,8765 running (8m) 75s ago 8m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 94b176253d9e 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (11m) 39s ago 11m 49.6M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 10511c159478 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (9m) 75s ago 9m 41.2M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 254ec67692aa 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi173 running (9m) 75s ago 9m 43.0M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 7eb0a02b2444 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (8m) 39s ago 8m 52.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b c66d1a4f17b5 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi153 running (8m) 75s ago 8m 56.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b e665ef106a91 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi173 running (7m) 75s ago 7m 56.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b cbeee5bfbc60 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (7m) 39s ago 7m 43.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8ffe887bc0a7 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (6m) 75s ago 6m 50.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 62124243dba8 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi173 running (6m) 75s ago 6m 53.3M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b fd64c0deaeae 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (5m) 39s ago 5m 49.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b ba51d04baec7 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (5m) 75s ago 5m 44.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 52b2c283214e 2024-03-16T12:20:58.528 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi173 running (4m) 75s ago 4m 45.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b d47cc4fc6391 2024-03-16T12:20:58.529 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (4m) 39s ago 4m 41.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 1075a1b07297 2024-03-16T12:20:58.529 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi153 running (3m) 75s ago 3m 45.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b a54555d7bd83 2024-03-16T12:20:58.529 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi173 running (3m) 75s ago 3m 49.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b faf99055eb2b 2024-03-16T12:20:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 msgr2=0x7fe3700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 0x7fe3700a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe38c076030 tx=0x7fe378009260 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 msgr2=0x7fe38c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00b070 tx=0x7fe37c004ce0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 shutdown_connections 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe38c10e8f0 0x7fe38c1a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c074d30 0x7fe38c19eb80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3700a2890 0x7fe3700a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.548+0000 7fe393e00700 1 --2- 172.21.15.98:0/3562090233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe38c076300 0x7fe38c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.549+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 >> 172.21.15.98:0/3562090233 conn(0x7fe38c0fd870 msgr2=0x7fe38c101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.549+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 shutdown_connections 2024-03-16T12:20:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:20:58.549+0000 7fe393e00700 1 -- 172.21.15.98:0/3562090233 wait complete. 2024-03-16T12:20:59.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T12:20:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:00.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:59 smithi098 ceph-mon[31633]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:00.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:20:59 smithi098 ceph-mon[31633]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:00.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:59 smithi153 ceph-mon[36901]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:00.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:20:59 smithi153 ceph-mon[36901]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:00.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:59 smithi173 ceph-mon[36632]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:00.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:20:59 smithi173 ceph-mon[36632]: pgmap v389: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:02.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:01 smithi098 ceph-mon[31633]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 -- 172.21.15.98:0/2939424339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec108950 msgr2=0x7fe3ec108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/2939424339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec108950 0x7fe3ec108d70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0099c0 tx=0x7fe3dc009cd0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 -- 172.21.15.98:0/2939424339 shutdown_connections 2024-03-16T12:21:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/2939424339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec108950 0x7fe3ec108d70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/2939424339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/2939424339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec109b50 0x7fe3ec109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 -- 172.21.15.98:0/2939424339 >> 172.21.15.98:0/2939424339 conn(0x7fe3ec077920 msgr2=0x7fe3ec079d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 -- 172.21.15.98:0/2939424339 shutdown_connections 2024-03-16T12:21:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.131+0000 7fe3f1a00700 1 -- 172.21.15.98:0/2939424339 wait complete. 2024-03-16T12:21:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.133+0000 7fe3f1a00700 1 Processor -- start 2024-03-16T12:21:02.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.133+0000 7fe3f1a00700 1 -- start start 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec109b50 0x7fe3ec19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec1a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec114580 con 0x7fe3ec109b50 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec114400 con 0x7fe3ec10a4f0 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3f1a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec114700 con 0x7fe3ec108950 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3eaa00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49524/0 (socket says 172.21.15.98:49524) 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3eaa00700 1 -- 172.21.15.98:0/3636065626 learned_addr learned my addr 172.21.15.98:0/3636065626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3eb400700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec1a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.134+0000 7fe3ea000700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec109b50 0x7fe3ec19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:02.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 -- 172.21.15.98:0/3636065626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 msgr2=0x7fe3ec1a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec1a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 -- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec109b50 msgr2=0x7fe3ec19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec109b50 0x7fe3ec19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 -- 172.21.15.98:0/3636065626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec1a6790 con 0x7fe3ec108950 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eb400700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec1a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3eaa00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00bee0 tx=0x7fe3dc00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc009e60 con 0x7fe3ec108950 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3dc00f460 con 0x7fe3ec108950 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec1a6a20 con 0x7fe3ec108950 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.135+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc017650 con 0x7fe3ec108950 2024-03-16T12:21:02.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.136+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec1a6ed0 con 0x7fe3ec108950 2024-03-16T12:21:02.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.137+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec068f90 con 0x7fe3ec108950 2024-03-16T12:21:02.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.138+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe3dc00f610 con 0x7fe3ec108950 2024-03-16T12:21:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.139+0000 7fe3e3400700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 0x7fe3d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.140+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe3dc0cfbb0 con 0x7fe3ec108950 2024-03-16T12:21:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.140+0000 7fe3ea000700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 0x7fe3d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.140+0000 7fe3ea000700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 0x7fe3d00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009c60 tx=0x7fe3d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.144+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3dc013070 con 0x7fe3ec108950 2024-03-16T12:21:02.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:01 smithi153 ceph-mon[36901]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:02.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:01 smithi173 ceph-mon[36632]: pgmap v390: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:02.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.376+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fe3ec062710 con 0x7fe3d00a2910 2024-03-16T12:21:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.393+0000 7fe3e3400700 1 -- 172.21.15.98:0/3636065626 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fe3ec062710 con 0x7fe3d00a2910 2024-03-16T12:21:02.375 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T12:21:02.375 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 79s ago 3m smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-03-16T12:21:02.376 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 79s ago 9m smithi098=x;smithi153=y;smithi173=z;count:3 2024-03-16T12:21:02.376 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 79s ago 9m smithi098:172.21.15.98=a;smithi153:172.21.15.153=b;smithi173:172.21.15.173=c;count:3 2024-03-16T12:21:02.376 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 79s ago - 2024-03-16T12:21:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 msgr2=0x7fe3d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 0x7fe3d00a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009c60 tx=0x7fe3d8009260 comp rx=0 tx=0).stop 2024-03-16T12:21:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 msgr2=0x7fe3ec19eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00bee0 tx=0x7fe3dc00bf10 comp rx=0 tx=0).stop 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 shutdown_connections 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe3d00a2910 0x7fe3d00a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3ec10a4f0 0x7fe3ec1a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec108950 0x7fe3ec19eb30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 --2- 172.21.15.98:0/3636065626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec109b50 0x7fe3ec19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.396+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 >> 172.21.15.98:0/3636065626 conn(0x7fe3ec077920 msgr2=0x7fe3ec10cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.397+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 shutdown_connections 2024-03-16T12:21:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:02.397+0000 7fe3f1a00700 1 -- 172.21.15.98:0/3636065626 wait complete. 2024-03-16T12:21:03.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T12:21:03.670 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:04.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:03 smithi098 ceph-mon[31633]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:04.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:03 smithi098 ceph-mon[31633]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:04.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:03 smithi153 ceph-mon[36901]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:04.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:03 smithi153 ceph-mon[36901]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:04.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:03 smithi173 ceph-mon[36632]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:04.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:03 smithi173 ceph-mon[36632]: pgmap v391: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:05.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 -- 172.21.15.98:0/3874074793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104150 msgr2=0x7f7c101045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:05.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 --2- 172.21.15.98:0/3874074793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104150 0x7f7c101045b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80099c0 tx=0x7f7bf8009cd0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 -- 172.21.15.98:0/3874074793 shutdown_connections 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 --2- 172.21.15.98:0/3874074793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104af0 0x7f7c1010c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 --2- 172.21.15.98:0/3874074793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c10106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 --2- 172.21.15.98:0/3874074793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104150 0x7f7c101045b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 -- 172.21.15.98:0/3874074793 >> 172.21.15.98:0/3874074793 conn(0x7f7c100ffaf0 msgr2=0x7f7c10101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 -- 172.21.15.98:0/3874074793 shutdown_connections 2024-03-16T12:21:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.984+0000 7f7c15000700 1 -- 172.21.15.98:0/3874074793 wait complete. 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.985+0000 7f7c15000700 1 Processor -- start 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.985+0000 7f7c15000700 1 -- start start 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 0x7f7c1019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c101a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c10110250 con 0x7f7c10104150 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101100d0 con 0x7f7c10104af0 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c15000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101103d0 con 0x7f7c10105c10 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c0de00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 0x7f7c1019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c07e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:05.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c0de00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 0x7f7c1019a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33976/0 (socket says 172.21.15.98:33976) 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c0e800700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c101a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c0de00700 1 -- 172.21.15.98:0/1848474707 learned_addr learned my addr 172.21.15.98:0/1848474707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.986+0000 7f7c07e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55716/0 (socket says 172.21.15.98:55716) 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 -- 172.21.15.98:0/1848474707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 msgr2=0x7f7c1019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 -- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 msgr2=0x7f7c1019a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 0x7f7c1019a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 -- 172.21.15.98:0/1848474707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c101a2470 con 0x7f7c10105c10 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c07e00700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c0e800700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c101a1dd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009f60 tx=0x7f7c08009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c080092e0 con 0x7f7c10105c10 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c08004bb0 con 0x7f7c10105c10 2024-03-16T12:21:05.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08004280 con 0x7f7c10105c10 2024-03-16T12:21:05.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c101a2700 con 0x7f7c10105c10 2024-03-16T12:21:05.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.987+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c101a2c40 con 0x7f7c10105c10 2024-03-16T12:21:05.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.988+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c10068f90 con 0x7f7c10105c10 2024-03-16T12:21:05.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.990+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c0801b030 con 0x7f7c10105c10 2024-03-16T12:21:05.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.991+0000 7f7c06a00700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 0x7f7bf40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:05.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.992+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c080ce820 con 0x7f7c10105c10 2024-03-16T12:21:05.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.992+0000 7f7c0de00700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 0x7f7bf40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:05.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.992+0000 7f7c0de00700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 0x7f7bf40a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009680 tx=0x7f7c00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:05.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:05.996+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c08022020 con 0x7f7c10105c10 2024-03-16T12:21:06.204 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:05 smithi098 ceph-mon[31633]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.222+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7c10062710 con 0x7f7bf40a2910 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.225+0000 7f7c06a00700 1 -- 172.21.15.98:0/1848474707 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f7c10062710 con 0x7f7bf40a2910 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stdout:smithi153 172.21.15.153 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 172.21.15.173 2024-03-16T12:21:06.207 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-03-16T12:21:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 msgr2=0x7f7bf40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 0x7f7bf40a4dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009680 tx=0x7f7c00009260 comp rx=0 tx=0).stop 2024-03-16T12:21:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 msgr2=0x7f7c101a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c101a1dd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009f60 tx=0x7f7c08009f90 comp rx=0 tx=0).stop 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 shutdown_connections 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c10105c10 0x7f7c101a1dd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c10104150 0x7f7c1019a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7bf40a2910 0x7f7bf40a4dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 --2- 172.21.15.98:0/1848474707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10104af0 0x7f7c1019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.228+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 >> 172.21.15.98:0/1848474707 conn(0x7f7c100ffaf0 msgr2=0x7f7c101089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.229+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 shutdown_connections 2024-03-16T12:21:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:06.229+0000 7f7c15000700 1 -- 172.21.15.98:0/1848474707 wait complete. 2024-03-16T12:21:06.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:05 smithi153 ceph-mon[36901]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:06.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:05 smithi173 ceph-mon[36632]: pgmap v392: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:07.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T12:21:07.517 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:07.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:07 smithi098 ceph-mon[31633]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:07.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:07 smithi098 ceph-mon[31633]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:21:08.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:07 smithi153 ceph-mon[36901]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:08.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:07 smithi153 ceph-mon[36901]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:21:08.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:07 smithi173 ceph-mon[36632]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:08.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:07 smithi173 ceph-mon[36632]: pgmap v393: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 -- 172.21.15.98:0/1235952175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c1060f0 msgr2=0x7ff06c10c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 --2- 172.21.15.98:0/1235952175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c1060f0 0x7ff06c10c9a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff06400b0a0 tx=0x7ff06400b3b0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 -- 172.21.15.98:0/1235952175 shutdown_connections 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 --2- 172.21.15.98:0/1235952175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c104550 0x7ff06c104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 --2- 172.21.15.98:0/1235952175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c1060f0 0x7ff06c10c9a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 --2- 172.21.15.98:0/1235952175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c105750 0x7ff06c105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 -- 172.21.15.98:0/1235952175 >> 172.21.15.98:0/1235952175 conn(0x7ff06c0ffaf0 msgr2=0x7ff06c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 -- 172.21.15.98:0/1235952175 shutdown_connections 2024-03-16T12:21:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.894+0000 7ff071a00700 1 -- 172.21.15.98:0/1235952175 wait complete. 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.895+0000 7ff071a00700 1 Processor -- start 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 -- start start 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 0x7ff06c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 0x7ff06c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c1060f0 0x7ff06c1a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c110170 con 0x7ff06c104550 2024-03-16T12:21:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c10fff0 con 0x7ff06c1060f0 2024-03-16T12:21:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff071a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c1102f0 con 0x7ff06c105750 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff06aa00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 0x7ff06c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff06a000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 0x7ff06c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff06b400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c1060f0 0x7ff06c1a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff06aa00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 0x7ff06c19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60746/0 (socket says 172.21.15.98:60746) 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.896+0000 7ff06aa00700 1 -- 172.21.15.98:0/1257690782 learned_addr learned my addr 172.21.15.98:0/1257690782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 -- 172.21.15.98:0/1257690782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c1060f0 msgr2=0x7ff06c1a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c1060f0 0x7ff06c1a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 -- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 msgr2=0x7ff06c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 0x7ff06c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 -- 172.21.15.98:0/1257690782 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c1a2290 con 0x7ff06c105750 2024-03-16T12:21:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff06a000700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 0x7ff06c19ab70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c9a0 tx=0x7ff05800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058014070 con 0x7ff06c105750 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff058004440 con 0x7ff06c105750 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05800eb10 con 0x7ff06c105750 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.897+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c1a2580 con 0x7ff06c105750 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.898+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c1a2ac0 con 0x7ff06c105750 2024-03-16T12:21:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.899+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c068f90 con 0x7ff06c105750 2024-03-16T12:21:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.900+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff058017070 con 0x7ff06c105750 2024-03-16T12:21:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.901+0000 7ff063400700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 0x7ff0500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.902+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0580ce730 con 0x7ff06c105750 2024-03-16T12:21:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.902+0000 7ff06aa00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 0x7ff0500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.903+0000 7ff06aa00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 0x7ff0500a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005950 tx=0x7ff05c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:09.906+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff05801e030 con 0x7ff06c105750 2024-03-16T12:21:10.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.134+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff06c062710 con 0x7ff0500a2890 2024-03-16T12:21:10.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.139+0000 7ff063400700 1 -- 172.21.15.98:0/1257690782 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff06c062710 con 0x7ff0500a2890 2024-03-16T12:21:10.120 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T12:21:10.121 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-16T12:21:10.121 INFO:teuthology.orchestra.run.smithi098.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T12:21:10.121 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T12:21:10.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.141+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 msgr2=0x7ff0500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:10.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.141+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 0x7ff0500a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005950 tx=0x7ff05c009260 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.141+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 msgr2=0x7ff06c19ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.141+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 0x7ff06c19ab70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c9a0 tx=0x7ff05800ccb0 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 shutdown_connections 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0500a2890 0x7ff0500a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06c1060f0 0x7ff06c1a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff06c104550 0x7ff06c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 --2- 172.21.15.98:0/1257690782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff06c105750 0x7ff06c19ab70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 >> 172.21.15.98:0/1257690782 conn(0x7ff06c0ffaf0 msgr2=0x7ff06c108980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 shutdown_connections 2024-03-16T12:21:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:10.142+0000 7ff071a00700 1 -- 172.21.15.98:0/1257690782 wait complete. 2024-03-16T12:21:10.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:09 smithi098 ceph-mon[31633]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:10.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:09 smithi153 ceph-mon[36901]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:10.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:09 smithi173 ceph-mon[36632]: pgmap v394: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:10.979 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T12:21:10.990 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:21:10.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T12:21:11.440 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:12.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:11 smithi098 ceph-mon[31633]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:12.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:11 smithi098 ceph-mon[31633]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:12.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:11 smithi153 ceph-mon[36901]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:12.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:11 smithi153 ceph-mon[36901]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:12.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:11 smithi173 ceph-mon[36632]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:21:12.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:11 smithi173 ceph-mon[36632]: pgmap v395: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 -- 172.21.15.98:0/2061590134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc4076300 msgr2=0x7f6bc4074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 --2- 172.21.15.98:0/2061590134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc4076300 0x7f6bc4074760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40098f0 tx=0x7f6bb4009c00 comp rx=0 tx=0).stop 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 -- 172.21.15.98:0/2061590134 shutdown_connections 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 --2- 172.21.15.98:0/2061590134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc4076300 0x7f6bc4074760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 --2- 172.21.15.98:0/2061590134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc410e900 0x7f6bc4110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 --2- 172.21.15.98:0/2061590134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4074d30 0x7f6bc40751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 -- 172.21.15.98:0/2061590134 >> 172.21.15.98:0/2061590134 conn(0x7f6bc40fd8e0 msgr2=0x7f6bc40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.687+0000 7f6bca000700 1 -- 172.21.15.98:0/2061590134 shutdown_connections 2024-03-16T12:21:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.688+0000 7f6bca000700 1 -- 172.21.15.98:0/2061590134 wait complete. 2024-03-16T12:21:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 Processor -- start 2024-03-16T12:21:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 -- start start 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4076300 0x7f6bc419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc410e900 0x7f6bc41a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4114530 con 0x7f6bc4076300 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc41143b0 con 0x7f6bc410e900 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bca000700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc41146b0 con 0x7f6bc4074d30 2024-03-16T12:21:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bc3400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bc3400700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50932/0 (socket says 172.21.15.98:50932) 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bc3400700 1 -- 172.21.15.98:0/1984191313 learned_addr learned my addr 172.21.15.98:0/1984191313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bc3e00700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc410e900 0x7f6bc41a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.689+0000 7f6bc2a00700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4076300 0x7f6bc419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 -- 172.21.15.98:0/1984191313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc410e900 msgr2=0x7f6bc41a6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc410e900 0x7f6bc41a6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 -- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4076300 msgr2=0x7f6bc419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4076300 0x7f6bc419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 -- 172.21.15.98:0/1984191313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc41a6810 con 0x7f6bc4074d30 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bc3400700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009230 tx=0x7f6bb4005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4025070 con 0x7f6bc4074d30 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bb400f950 con 0x7f6bc4074d30 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc41a6aa0 con 0x7f6bc4074d30 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.690+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4003c00 con 0x7f6bc4074d30 2024-03-16T12:21:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.691+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc41a6ef0 con 0x7f6bc4074d30 2024-03-16T12:21:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.692+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc4068f90 con 0x7f6bc4074d30 2024-03-16T12:21:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.693+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6bb4028070 con 0x7f6bc4074d30 2024-03-16T12:21:13.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.695+0000 7f6bbbe00700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 0x7f6ba80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:13.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.695+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6bb40cf640 con 0x7f6bc4074d30 2024-03-16T12:21:13.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.695+0000 7f6bc2a00700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 0x7f6ba80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:13.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.696+0000 7f6bc2a00700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 0x7f6ba80a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4076030 tx=0x7f6bb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:13.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.699+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6bb405c740 con 0x7f6bc4074d30 2024-03-16T12:21:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.972+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f6bc4062f80 con 0x7f6bc4074d30 2024-03-16T12:21:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.972+0000 7f6bbbe00700 1 -- 172.21.15.98:0/1984191313 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3052 (secure 0 0 0) 0x7f6bb4013030 con 0x7f6bc4074d30 2024-03-16T12:21:13.954 INFO:teuthology.orchestra.run.smithi098.stdout:e23 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 23 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-16T12:18:40.454968+0000 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-03-16T12:19:07.477601+0000 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-03-16T12:21:13.955 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 3 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24599,1=24553,2=24542} 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3,4] 2024-03-16T12:21:13.956 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:qdb_cluster leader: 24599 members: 24553,24599,24542 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24599} state up:active seq 3 addr [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{0:24529} state up:standby-replay seq 1 addr [v2:172.21.15.173:6836/4083501894,v1:172.21.15.173:6837/4083501894] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{1:24553} state up:active seq 12 addr [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{1:24517} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/2799668647,v1:172.21.15.173:6835/2799668647] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{2:24542} state up:active seq 13 addr [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{2:24512} state up:standby-replay seq 1 addr [v2:172.21.15.153:6834/4013234440,v1:172.21.15.153:6835/4013234440] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-03-16T12:21:13.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14397} state up:standby seq 1 addr [v2:172.21.15.98:6834/211337722,v1:172.21.15.98:6835/211337722] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{-1:14463} state up:standby seq 1 addr [v2:172.21.15.98:6838/1905625242,v1:172.21.15.98:6839/1905625242] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{-1:14481} state up:standby seq 1 addr [v2:172.21.15.98:6840/1343544864,v1:172.21.15.98:6841/1343544864] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{-1:24448} state up:standby seq 1 addr [v2:172.21.15.173:6832/307291521,v1:172.21.15.173:6833/307291521] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{-1:24467} state up:standby seq 1 addr [v2:172.21.15.153:6832/3502700465,v1:172.21.15.153:6833/3502700465] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.958 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:24490} state up:standby seq 1 addr [v2:172.21.15.98:6836/3487564371,v1:172.21.15.98:6837/3487564371] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.975+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 msgr2=0x7f6ba80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.975+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 0x7f6ba80a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4076030 tx=0x7f6bb0009260 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.975+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 msgr2=0x7f6bc419ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.975+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009230 tx=0x7f6bb4005250 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 shutdown_connections 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ba80a2890 0x7f6ba80a4d50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bc410e900 0x7f6bc41a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bc4074d30 0x7f6bc419ec00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 --2- 172.21.15.98:0/1984191313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc4076300 0x7f6bc419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 >> 172.21.15.98:0/1984191313 conn(0x7f6bc40fd8e0 msgr2=0x7f6bc4101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 shutdown_connections 2024-03-16T12:21:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:13.976+0000 7f6bca000700 1 -- 172.21.15.98:0/1984191313 wait complete. 2024-03-16T12:21:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:21:14.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:13 smithi098 ceph-mon[31633]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:14.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:13 smithi153 ceph-mon[36901]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:14.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:13 smithi173 ceph-mon[36632]: pgmap v396: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:14.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T12:21:15.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:14 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1984191313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T12:21:15.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:14 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1984191313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T12:21:15.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:14 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1984191313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T12:21:15.321 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:16.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:15 smithi098 ceph-mon[31633]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:16.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:15 smithi153 ceph-mon[36901]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:16.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:15 smithi173 ceph-mon[36632]: pgmap v397: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 -- 172.21.15.98:0/3403232477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103fe0 msgr2=0x7f001010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 --2- 172.21.15.98:0/3403232477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103fe0 0x7f001010a890 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f000800b0a0 tx=0x7f000800b3b0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 -- 172.21.15.98:0/3403232477 shutdown_connections 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 --2- 172.21.15.98:0/3403232477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103fe0 0x7f001010a890 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 --2- 172.21.15.98:0/3403232477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 0x7f00101009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 --2- 172.21.15.98:0/3403232477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103640 0x7f0010103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 -- 172.21.15.98:0/3403232477 >> 172.21.15.98:0/3403232477 conn(0x7f00100fbf40 msgr2=0x7f00100fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 -- 172.21.15.98:0/3403232477 shutdown_connections 2024-03-16T12:21:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.705+0000 7f0017600700 1 -- 172.21.15.98:0/3403232477 wait complete. 2024-03-16T12:21:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.706+0000 7f0017600700 1 Processor -- start 2024-03-16T12:21:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.706+0000 7f0017600700 1 -- start start 2024-03-16T12:21:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 0x7f0010198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103fe0 0x7f001019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f001010deb0 con 0x7f00101005c0 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f001010dd30 con 0x7f0010103640 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0017600700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f001010e030 con 0x7f0010103fe0 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f000fe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0014c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 0x7f0010198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f000fe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51034/0 (socket says 172.21.15.98:51034) 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f000fe00700 1 -- 172.21.15.98:0/2193375461 learned_addr learned my addr 172.21.15.98:0/2193375461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.707+0000 7f0015600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103fe0 0x7f001019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 -- 172.21.15.98:0/2193375461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103fe0 msgr2=0x7f001019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103fe0 0x7f001019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 -- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 msgr2=0x7f0010198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 0x7f0010198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 -- 172.21.15.98:0/2193375461 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00101a00a0 con 0x7f0010103640 2024-03-16T12:21:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000fe00700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e9a0 tx=0x7efffc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc00ca10 con 0x7f0010103640 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efffc00cbb0 con 0x7f0010103640 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc018710 con 0x7f0010103640 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00101a0390 con 0x7f0010103640 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.708+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00101a0920 con 0x7f0010103640 2024-03-16T12:21:17.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.709+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0010068f90 con 0x7f0010103640 2024-03-16T12:21:17.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.712+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efffc047a60 con 0x7f0010103640 2024-03-16T12:21:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.713+0000 7f000d600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 0x7efff80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.713+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efffc0ceef0 con 0x7f0010103640 2024-03-16T12:21:17.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.713+0000 7f0014c00700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 0x7efff80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:17.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.714+0000 7f0014c00700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 0x7efff80a4dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0010069ce0 tx=0x7f0004009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:17.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.720+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efffc09bbd0 con 0x7f0010103640 2024-03-16T12:21:17.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.945+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0010062ee0 con 0x7f0010103640 2024-03-16T12:21:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.947+0000 7f000d600700 1 -- 172.21.15.98:0/2193375461 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5601 (secure 0 0 0) 0x7efffc09b4d0 con 0x7f0010103640 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 91 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-16T12:09:34.983809+0000 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-03-16T12:19:44.538706+0000 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 27 2024-03-16T12:21:17.929 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release squid 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/3304246268,v1:172.21.15.98:6803/3304246268] [v2:172.21.15.98:6804/3304246268,v1:172.21.15.98:6805/3304246268] exists,up 464e1266-b151-4bf9-82b5-398b1aa8d1db 2024-03-16T12:21:17.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/4082140415,v1:172.21.15.153:6801/4082140415] [v2:172.21.15.153:6802/4082140415,v1:172.21.15.153:6803/4082140415] exists,up eb71b246-921e-49df-9e6d-08cd3431f968 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/2055802330,v1:172.21.15.173:6801/2055802330] [v2:172.21.15.173:6802/2055802330,v1:172.21.15.173:6803/2055802330] exists,up 1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/2502573186,v1:172.21.15.98:6811/2502573186] [v2:172.21.15.98:6812/2502573186,v1:172.21.15.98:6813/2502573186] exists,up 5e1de00e-5fc6-4a53-9cfa-99cede3449aa 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/3184279467,v1:172.21.15.153:6809/3184279467] [v2:172.21.15.153:6810/3184279467,v1:172.21.15.153:6811/3184279467] exists,up b864c756-758e-4799-a35d-b83d204b3727 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/3808045803,v1:172.21.15.173:6809/3808045803] [v2:172.21.15.173:6810/3808045803,v1:172.21.15.173:6811/3808045803] exists,up c31daad0-fd36-4b7c-98d3-64e2057c4779 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/3005930753,v1:172.21.15.98:6819/3005930753] [v2:172.21.15.98:6820/3005930753,v1:172.21.15.98:6821/3005930753] exists,up 229b1cd5-870e-4b6c-b3af-243b19a68157 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/838772754,v1:172.21.15.153:6817/838772754] [v2:172.21.15.153:6818/838772754,v1:172.21.15.153:6819/838772754] exists,up 615a84ba-ee1b-4ea0-92c9-0211098d2215 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/785354592,v1:172.21.15.173:6817/785354592] [v2:172.21.15.173:6818/785354592,v1:172.21.15.173:6819/785354592] exists,up fa255236-2797-4a72-b784-65cc6c731190 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/3340262807,v1:172.21.15.98:6827/3340262807] [v2:172.21.15.98:6828/3340262807,v1:172.21.15.98:6829/3340262807] exists,up cc2c29f0-167c-40d8-8ca6-4758e66f435c 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/2757700029,v1:172.21.15.153:6825/2757700029] [v2:172.21.15.153:6826/2757700029,v1:172.21.15.153:6827/2757700029] exists,up deab615a-9d50-4480-ac75-93033b168f40 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/567612982,v1:172.21.15.173:6825/567612982] [v2:172.21.15.173:6826/567612982,v1:172.21.15.173:6827/567612982] exists,up 95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.b [8,11] 2024-03-16T12:21:17.931 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.13 [2,11] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.0 [4,8] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.21 [5,9] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.2c [4,9] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.36 [4,9] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.3 [5,10] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.6 [9,11] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.16 [9,8] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.18 [9,8] 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/3396936170 expires 2024-03-17T12:10:42.632912+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/3396936170 expires 2024-03-17T12:10:42.632912+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/352508062 expires 2024-03-17T12:10:42.632912+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/653650228 expires 2024-03-17T12:10:08.286891+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/719012091 expires 2024-03-17T12:10:42.632912+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/3029136477 expires 2024-03-17T12:10:08.286891+0000 2024-03-16T12:21:17.932 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/196846375 expires 2024-03-17T12:10:42.632912+0000 2024-03-16T12:21:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3496387635 expires 2024-03-17T12:10:08.286891+0000 2024-03-16T12:21:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3016831032 expires 2024-03-17T12:10:08.286891+0000 2024-03-16T12:21:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/3029136477 expires 2024-03-17T12:10:08.286891+0000 2024-03-16T12:21:17.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 msgr2=0x7efff80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 0x7efff80a4dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0010069ce0 tx=0x7f0004009260 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 msgr2=0x7f0010198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e9a0 tx=0x7efffc00ecb0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 shutdown_connections 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0010103fe0 0x7f001019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00101005c0 0x7f0010198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efff80a2910 0x7efff80a4dd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 --2- 172.21.15.98:0/2193375461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0010103640 0x7f0010198980 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 >> 172.21.15.98:0/2193375461 conn(0x7f00100fbf40 msgr2=0x7f0010106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 shutdown_connections 2024-03-16T12:21:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:17.950+0000 7f0017600700 1 -- 172.21.15.98:0/2193375461 wait complete. 2024-03-16T12:21:18.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:17 smithi098 ceph-mon[31633]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:17 smithi153 ceph-mon[36901]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:18.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:17 smithi173 ceph-mon[36632]: pgmap v398: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:18.925 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T12:21:18.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 -- 172.21.15.98:0/453039216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 msgr2=0x7f9ff8103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/453039216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8103c30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8009900 tx=0x7f9fe802bc40 comp rx=0 tx=0).stop 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 -- 172.21.15.98:0/453039216 shutdown_connections 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/453039216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff81053c0 0x7f9ff810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/453039216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff8104a00 0x7f9ff8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/453039216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8103c30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 -- 172.21.15.98:0/453039216 >> 172.21.15.98:0/453039216 conn(0x7f9ff80fed80 msgr2=0x7f9ff81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 -- 172.21.15.98:0/453039216 shutdown_connections 2024-03-16T12:21:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.174+0000 7f9ff7e00640 1 -- 172.21.15.98:0/453039216 wait complete. 2024-03-16T12:21:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 Processor -- start 2024-03-16T12:21:19.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 -- start start 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 0x7f9ff819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff81053c0 0x7f9ff81a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810f490 con 0x7f9ff81053c0 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810f310 con 0x7f9ff8103810 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.175+0000 7f9ff7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810f610 con 0x7f9ff8104a00 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9ff6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9ff6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51046/0 (socket says 172.21.15.98:51046) 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9ff6a00640 1 -- 172.21.15.98:0/2612654450 learned_addr learned my addr 172.21.15.98:0/2612654450 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 0x7f9ff819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9ff7400640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff81053c0 0x7f9ff81a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 -- 172.21.15.98:0/2612654450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 msgr2=0x7f9ff8199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 -- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff81053c0 msgr2=0x7f9ff81a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff81053c0 0x7f9ff81a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 -- 172.21.15.98:0/2612654450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff81a1700 con 0x7f9ff8104a00 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9ff6a00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fefe00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 0x7f9ff819a020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000ccc0 con 0x7f9ff8104a00 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.176+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0010910 con 0x7f9ff8104a00 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.177+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff81a19f0 con 0x7f9ff8104a00 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.177+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0018960 con 0x7f9ff8104a00 2024-03-16T12:21:19.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.177+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff81a1f90 con 0x7f9ff8104a00 2024-03-16T12:21:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.178+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff810ab30 con 0x7f9ff8104a00 2024-03-16T12:21:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.179+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9fe0010ac0 con 0x7f9ff8104a00 2024-03-16T12:21:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.181+0000 7f9fef400640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.181+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9fe00cf0d0 con 0x7f9ff8104a00 2024-03-16T12:21:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.181+0000 7f9ff6a00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.182+0000 7f9ff6a00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9fe802c8f0 tx=0x7f9fe8005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:19.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.185+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fe009b7c0 con 0x7f9ff8104a00 2024-03-16T12:21:19.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:18 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2193375461' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T12:21:19.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:18 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2193375461' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T12:21:19.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:18 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2193375461' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T12:21:19.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.385+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ff80623e0 con 0x7f9ff8104a00 2024-03-16T12:21:19.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.387+0000 7f9fef400640 1 -- 172.21.15.98:0/2612654450 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f9fe009f860 con 0x7f9ff8104a00 2024-03-16T12:21:19.369 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:19.369 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 msgr2=0x7f9fdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9fe802c8f0 tx=0x7f9fe8005b50 comp rx=0 tx=0).stop 2024-03-16T12:21:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 msgr2=0x7f9ff819a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 0x7f9ff819a020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 shutdown_connections 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff81053c0 0x7f9ff81a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ff8104a00 0x7f9ff819a020 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 --2- 172.21.15.98:0/2612654450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ff8103810 0x7f9ff8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.389+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 >> 172.21.15.98:0/2612654450 conn(0x7f9ff80fed80 msgr2=0x7f9ff8107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.390+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 shutdown_connections 2024-03-16T12:21:19.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.390+0000 7f9ff7e00640 1 -- 172.21.15.98:0/2612654450 wait complete. 2024-03-16T12:21:19.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:21:19.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:21:19.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 -- 172.21.15.98:0/1310180862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0103a20 msgr2=0x7f96b0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 --2- 172.21.15.98:0/1310180862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0103a20 0x7f96b0103e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009900 tx=0x7f96a002bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 -- 172.21.15.98:0/1310180862 shutdown_connections 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 --2- 172.21.15.98:0/1310180862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b01055d0 0x7f96b010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 --2- 172.21.15.98:0/1310180862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0104c10 0x7f96b0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 --2- 172.21.15.98:0/1310180862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b0103a20 0x7f96b0103e40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 -- 172.21.15.98:0/1310180862 >> 172.21.15.98:0/1310180862 conn(0x7f96b00fef90 msgr2=0x7f96b01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 -- 172.21.15.98:0/1310180862 shutdown_connections 2024-03-16T12:21:19.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.624+0000 7f96b7800640 1 -- 172.21.15.98:0/1310180862 wait complete. 2024-03-16T12:21:19.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 Processor -- start 2024-03-16T12:21:19.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 -- start start 2024-03-16T12:21:19.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b0104c10 0x7f96b019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 0x7f96b01a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010f6e0 con 0x7f96b0104c10 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010f560 con 0x7f96b0103a20 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.625+0000 7f96b7800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010f860 con 0x7f96b01055d0 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b4e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b4e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51062/0 (socket says 172.21.15.98:51062) 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b4e00640 1 -- 172.21.15.98:0/1834092185 learned_addr learned my addr 172.21.15.98:0/1834092185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 0x7f96b01a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96a7e00640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b0104c10 0x7f96b019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 -- 172.21.15.98:0/1834092185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 msgr2=0x7f96b0199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 -- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b0104c10 msgr2=0x7f96b019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b0104c10 0x7f96b019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 -- 172.21.15.98:0/1834092185 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b01a1970 con 0x7f96b01055d0 2024-03-16T12:21:19.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b5800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 0x7f96b01a1300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009fd0 tx=0x7f96a800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800ce20 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8005020 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b01a1c60 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.626+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8010720 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.627+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01a21b0 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.628+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b010ad40 con 0x7f96b01055d0 2024-03-16T12:21:19.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.629+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f96a8005540 con 0x7f96b01055d0 2024-03-16T12:21:19.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.631+0000 7f96ae000640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:19.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.631+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96a80cf400 con 0x7f96b01055d0 2024-03-16T12:21:19.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.631+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:19.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.632+0000 7f96b4e00640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f96a0006010 tx=0x7f96a002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:19.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.635+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96a809c140 con 0x7f96b01055d0 2024-03-16T12:21:19.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.832+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96b00623e0 con 0x7f96b01055d0 2024-03-16T12:21:19.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.833+0000 7f96ae000640 1 -- 172.21.15.98:0/1834092185 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f96a809ba40 con 0x7f96b01055d0 2024-03-16T12:21:19.815 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:21:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.835+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 msgr2=0x7f96940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.835+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f96a0006010 tx=0x7f96a002df90 comp rx=0 tx=0).stop 2024-03-16T12:21:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.835+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 msgr2=0x7f96b01a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.835+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 0x7f96b01a1300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009fd0 tx=0x7f96a800ef10 comp rx=0 tx=0).stop 2024-03-16T12:21:19.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 shutdown_connections 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f96940a2880 0x7f96940a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b01055d0 0x7f96b01a1300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b0104c10 0x7f96b019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 --2- 172.21.15.98:0/1834092185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b0103a20 0x7f96b0199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 >> 172.21.15.98:0/1834092185 conn(0x7f96b00fef90 msgr2=0x7f96b0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 shutdown_connections 2024-03-16T12:21:19.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:19.836+0000 7f96b7800640 1 -- 172.21.15.98:0/1834092185 wait complete. 2024-03-16T12:21:19.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 -- 172.21.15.98:0/4175320007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 msgr2=0x7f4170105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 --2- 172.21.15.98:0/4175320007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f4170105070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4158009900 tx=0x7f415802bc40 comp rx=0 tx=0).stop 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 -- 172.21.15.98:0/4175320007 shutdown_connections 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 --2- 172.21.15.98:0/4175320007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f417010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 --2- 172.21.15.98:0/4175320007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f4170105070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 --2- 172.21.15.98:0/4175320007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 -- 172.21.15.98:0/4175320007 >> 172.21.15.98:0/4175320007 conn(0x7f41700fef90 msgr2=0x7f41701013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 -- 172.21.15.98:0/4175320007 shutdown_connections 2024-03-16T12:21:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.066+0000 7f4176400640 1 -- 172.21.15.98:0/4175320007 wait complete. 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.067+0000 7f4176400640 1 Processor -- start 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.067+0000 7f4176400640 1 -- start start 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f417019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f41701a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f417010f6b0 con 0x7f4170104bf0 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f417010f530 con 0x7f41701055b0 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4176400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f417010f830 con 0x7f4170103a00 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f41701a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f4166a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f417019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48610/0 (socket says 172.21.15.98:48610) 2024-03-16T12:21:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 -- 172.21.15.98:0/1814505248 learned_addr learned my addr 172.21.15.98:0/1814505248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416fe00640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f41701a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51076/0 (socket says 172.21.15.98:51076) 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 -- 172.21.15.98:0/1814505248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 msgr2=0x7f41701a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f41701a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 -- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 msgr2=0x7f417019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f417019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.068+0000 7f416f400640 1 -- 172.21.15.98:0/1814505248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41701a1900 con 0x7f4170103a00 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f416f400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9c0 tx=0x7f416000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416000cde0 con 0x7f4170103a00 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4160010910 con 0x7f4170103a00 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4160018a00 con 0x7f4170103a00 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41701a1bf0 con 0x7f4170103a00 2024-03-16T12:21:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.069+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41701a20f0 con 0x7f4170103a00 2024-03-16T12:21:20.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.070+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f417010ad20 con 0x7f4170103a00 2024-03-16T12:21:20.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.072+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f416001f020 con 0x7f4170103a00 2024-03-16T12:21:20.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.073+0000 7f416cc00640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 0x7f41540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.073+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f41600cefb0 con 0x7f4170103a00 2024-03-16T12:21:20.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.073+0000 7f4166a00640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 0x7f41540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.074+0000 7f4166a00640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 0x7f41540a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f415802c250 tx=0x7f415802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.077+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f416009bc90 con 0x7f4170103a00 2024-03-16T12:21:20.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:19 smithi098 ceph-mon[31633]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T12:21:20.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:19 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2612654450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:20.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:19 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1834092185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:20.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:19 smithi153 ceph-mon[36901]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T12:21:20.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:19 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2612654450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:20.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:19 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1834092185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:20.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.277+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41700624b0 con 0x7f4170103a00 2024-03-16T12:21:20.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.278+0000 7f416cc00640 1 -- 172.21.15.98:0/1814505248 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f416009b590 con 0x7f4170103a00 2024-03-16T12:21:20.259 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.280+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 msgr2=0x7f41540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.280+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 0x7f41540a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f415802c250 tx=0x7f415802cf90 comp rx=0 tx=0).stop 2024-03-16T12:21:20.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.280+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 msgr2=0x7f4170199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.280+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9c0 tx=0x7f416000ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 shutdown_connections 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41540a2880 0x7f41540a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41701055b0 0x7f41701a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4170104bf0 0x7f417019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 --2- 172.21.15.98:0/1814505248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4170103a00 0x7f4170199cf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 >> 172.21.15.98:0/1814505248 conn(0x7f41700fef90 msgr2=0x7f4170107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 shutdown_connections 2024-03-16T12:21:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.281+0000 7f4176400640 1 -- 172.21.15.98:0/1814505248 wait complete. 2024-03-16T12:21:20.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:21:20.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:19 smithi173 ceph-mon[36632]: pgmap v399: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-16T12:21:20.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:19 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2612654450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:20.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:19 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1834092185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 -- 172.21.15.98:0/347968601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9068108ff0 msgr2=0x7f9068109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 --2- 172.21.15.98:0/347968601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9068108ff0 0x7f9068109470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9054009900 tx=0x7f905402bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 -- 172.21.15.98:0/347968601 shutdown_connections 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 --2- 172.21.15.98:0/347968601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681099b0 0x7f9068110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 --2- 172.21.15.98:0/347968601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9068108ff0 0x7f9068109470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 --2- 172.21.15.98:0/347968601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068107e00 0x7f9068108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 -- 172.21.15.98:0/347968601 >> 172.21.15.98:0/347968601 conn(0x7f9068076da0 msgr2=0x7f90680791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 -- 172.21.15.98:0/347968601 shutdown_connections 2024-03-16T12:21:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.512+0000 7f906ec00640 1 -- 172.21.15.98:0/347968601 wait complete. 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.513+0000 7f906ec00640 1 Processor -- start 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 -- start start 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 0x7f906819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9068113a80 con 0x7f9068107e00 2024-03-16T12:21:20.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9068113900 con 0x7f90681099b0 2024-03-16T12:21:20.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9068113c00 con 0x7f9068108ff0 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f9067400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51104/0 (socket says 172.21.15.98:51104) 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f9067400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48632/0 (socket says 172.21.15.98:48632) 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f906cc00640 1 -- 172.21.15.98:0/2923045913 learned_addr learned my addr 172.21.15.98:0/2923045913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.514+0000 7f9067e00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 0x7f906819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 -- 172.21.15.98:0/2923045913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 msgr2=0x7f906819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 -- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 msgr2=0x7f906819e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 0x7f906819e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 -- 172.21.15.98:0/2923045913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90681a5d70 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f9067400640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f9067e00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 0x7f906819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906cc00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9060009fd0 tx=0x7f906000b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060017070 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9060003e50 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.515+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90681a6060 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.516+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90681a6580 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.516+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600106c0 con 0x7f90681099b0 2024-03-16T12:21:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.517+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90480052c0 con 0x7f90681099b0 2024-03-16T12:21:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.520+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9060003970 con 0x7f90681099b0 2024-03-16T12:21:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.521+0000 7f9064c00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 0x7f904c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.521+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f90600cea10 con 0x7f90681099b0 2024-03-16T12:21:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.521+0000 7f9067e00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 0x7f904c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.522+0000 7f9067e00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 0x7f904c0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9068108e50 tx=0x7f9058005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.523+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f906009b770 con 0x7f90681099b0 2024-03-16T12:21:20.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.726+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9048005880 con 0x7f90681099b0 2024-03-16T12:21:20.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.727+0000 7f9064c00640 1 -- 172.21.15.98:0/2923045913 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f906009b070 con 0x7f90681099b0 2024-03-16T12:21:20.708 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.729+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 msgr2=0x7f904c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.729+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 0x7f904c0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9068108e50 tx=0x7f9058005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.729+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 msgr2=0x7f90681a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.729+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9060009fd0 tx=0x7f906000b260 comp rx=0 tx=0).stop 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 shutdown_connections 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f904c0a2880 0x7f904c0a4d40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90681099b0 0x7f90681a5700 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9068108ff0 0x7f906819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 --2- 172.21.15.98:0/2923045913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9068107e00 0x7f906819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 >> 172.21.15.98:0/2923045913 conn(0x7f9068076da0 msgr2=0x7f906810c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 shutdown_connections 2024-03-16T12:21:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.730+0000 7f906ec00640 1 -- 172.21.15.98:0/2923045913 wait complete. 2024-03-16T12:21:20.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 -- 172.21.15.98:0/4167221522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc0757a0 msgr2=0x7f6cfc073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 --2- 172.21.15.98:0/4167221522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc073bf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009900 tx=0x7f6cec02bcd0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 -- 172.21.15.98:0/4167221522 shutdown_connections 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 --2- 172.21.15.98:0/4167221522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc109570 0x7f6cfc10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 --2- 172.21.15.98:0/4167221522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 --2- 172.21.15.98:0/4167221522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc073bf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 -- 172.21.15.98:0/4167221522 >> 172.21.15.98:0/4167221522 conn(0x7f6cfc0fcc50 msgr2=0x7f6cfc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 -- 172.21.15.98:0/4167221522 shutdown_connections 2024-03-16T12:21:20.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.963+0000 7f6cfb200640 1 -- 172.21.15.98:0/4167221522 wait complete. 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.964+0000 7f6cfb200640 1 Processor -- start 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 -- start start 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc109570 0x7f6cfc1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10f520 con 0x7f6cfc0757a0 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10f3a0 con 0x7f6cfc0741c0 2024-03-16T12:21:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10f6a0 con 0x7f6cfc109570 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cf9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cf9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51120/0 (socket says 172.21.15.98:51120) 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cf9e00640 1 -- 172.21.15.98:0/3793455698 learned_addr learned my addr 172.21.15.98:0/3793455698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cf9400640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.965+0000 7f6cfa800640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc109570 0x7f6cfc1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 -- 172.21.15.98:0/3793455698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc109570 msgr2=0x7f6cfc1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc109570 0x7f6cfc1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 -- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0757a0 msgr2=0x7f6cfc19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 -- 172.21.15.98:0/3793455698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc1a1810 con 0x7f6cfc0741c0 2024-03-16T12:21:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf9e00640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005ab0 tx=0x7f6cec02c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec041070 con 0x7f6cfc0741c0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec004450 con 0x7f6cfc0741c0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc1a1aa0 con 0x7f6cfc0741c0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec003e20 con 0x7f6cfc0741c0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.966+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc1a1f90 con 0x7f6cfc0741c0 2024-03-16T12:21:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.967+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc108330 con 0x7f6cfc0741c0 2024-03-16T12:21:20.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.969+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6cec044070 con 0x7f6cfc0741c0 2024-03-16T12:21:20.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.970+0000 7f6cf2a00640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 0x7f6ce00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:20.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.970+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6cec0f0880 con 0x7f6cfc0741c0 2024-03-16T12:21:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.970+0000 7f6cf9400640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 0x7f6ce00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.971+0000 7f6cf9400640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 0x7f6ce00a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc19b100 tx=0x7f6ce8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:20.974+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cec0bd5f0 con 0x7f6cfc0741c0 2024-03-16T12:21:20.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:20 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1814505248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:20.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:20 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:21:20.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:20 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2923045913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.192+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6cfc02d1b0 con 0x7f6cfc0741c0 2024-03-16T12:21:21.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.193+0000 7f6cf2a00640 1 -- 172.21.15.98:0/3793455698 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f6cec0bcef0 con 0x7f6cfc0741c0 2024-03-16T12:21:21.174 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:21.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 msgr2=0x7f6ce00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 0x7f6ce00a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc19b100 tx=0x7f6ce8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:21.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 msgr2=0x7f6cfc199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cec005ab0 tx=0x7f6cec02c6e0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 shutdown_connections 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ce00a2800 0x7f6ce00a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cfc109570 0x7f6cfc1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cfc0757a0 0x7f6cfc19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 --2- 172.21.15.98:0/3793455698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cfc0741c0 0x7f6cfc199bf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 >> 172.21.15.98:0/3793455698 conn(0x7f6cfc0fcc50 msgr2=0x7f6cfc105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 shutdown_connections 2024-03-16T12:21:21.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.195+0000 7f6cfb200640 1 -- 172.21.15.98:0/3793455698 wait complete. 2024-03-16T12:21:21.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:21:21.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:20 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1814505248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:21.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:20 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:21:21.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:20 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2923045913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:21.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:20 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1814505248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:21.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:20 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:21:21.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:20 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2923045913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:21.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 -- 172.21.15.98:0/633522720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d80741c0 msgr2=0x7f15d8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 --2- 172.21.15.98:0/633522720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d80741c0 0x7f15d8074620 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009900 tx=0x7f15c002bcc0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 -- 172.21.15.98:0/633522720 shutdown_connections 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 --2- 172.21.15.98:0/633522720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d81094e0 0x7f15d810b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 --2- 172.21.15.98:0/633522720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d80741c0 0x7f15d8074620 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 --2- 172.21.15.98:0/633522720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80757a0 0x7f15d8073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 -- 172.21.15.98:0/633522720 >> 172.21.15.98:0/633522720 conn(0x7f15d80fcbc0 msgr2=0x7f15d80fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 -- 172.21.15.98:0/633522720 shutdown_connections 2024-03-16T12:21:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.398+0000 7f15d7e00640 1 -- 172.21.15.98:0/633522720 wait complete. 2024-03-16T12:21:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.399+0000 7f15d7e00640 1 Processor -- start 2024-03-16T12:21:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.399+0000 7f15d7e00640 1 -- start start 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d80757a0 0x7f15d819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15d810f4b0 con 0x7f15d80757a0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15d810f330 con 0x7f15d81094e0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15d810f630 con 0x7f15d80741c0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15ce000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d80757a0 0x7f15d819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48670/0 (socket says 172.21.15.98:48670) 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d6a00640 1 -- 172.21.15.98:0/3022412735 learned_addr learned my addr 172.21.15.98:0/3022412735 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.400+0000 7f15d7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51130/0 (socket says 172.21.15.98:51130) 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 -- 172.21.15.98:0/3022412735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 msgr2=0x7f15d81a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 -- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d80757a0 msgr2=0x7f15d819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d80757a0 0x7f15d819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 -- 172.21.15.98:0/3022412735 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d81a16e0 con 0x7f15d80741c0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d7400640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d6a00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f15c800eb90 tx=0x7f15c800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8004b20 con 0x7f15d80741c0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15c8004cc0 con 0x7f15d80741c0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d81a19d0 con 0x7f15d80741c0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8010720 con 0x7f15d80741c0 2024-03-16T12:21:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.401+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d81a1ef0 con 0x7f15d80741c0 2024-03-16T12:21:21.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.402+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d81082a0 con 0x7f15d80741c0 2024-03-16T12:21:21.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.404+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f15c8005020 con 0x7f15d80741c0 2024-03-16T12:21:21.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.405+0000 7f15dca00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 0x7f15bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.406+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f15c80cf380 con 0x7f15d80741c0 2024-03-16T12:21:21.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.406+0000 7f15ce000640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 0x7f15bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.406+0000 7f15ce000640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 0x7f15bc0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f15c0005ab0 tx=0x7f15c002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:21.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.409+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15c809c060 con 0x7f15d80741c0 2024-03-16T12:21:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.649+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15d802d570 con 0x7f15d80741c0 2024-03-16T12:21:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.650+0000 7f15dca00640 1 -- 172.21.15.98:0/3022412735 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7f15c809b960 con 0x7f15d80741c0 2024-03-16T12:21:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:21.631 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:21:21.632 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 msgr2=0x7f15bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 0x7f15bc0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f15c0005ab0 tx=0x7f15c002df90 comp rx=0 tx=0).stop 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 msgr2=0x7f15d8199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f15c800eb90 tx=0x7f15c800ef90 comp rx=0 tx=0).stop 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 shutdown_connections 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f15bc0a2880 0x7f15bc0a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15d81094e0 0x7f15d81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d80757a0 0x7f15d819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 --2- 172.21.15.98:0/3022412735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d80741c0 0x7f15d8199ac0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 >> 172.21.15.98:0/3022412735 conn(0x7f15d80fcbc0 msgr2=0x7f15d8105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:21.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.652+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 shutdown_connections 2024-03-16T12:21:21.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.653+0000 7f15d7e00640 1 -- 172.21.15.98:0/3022412735 wait complete. 2024-03-16T12:21:21.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 -- 172.21.15.98:0/489279410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81029e0 msgr2=0x7f41d8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 --2- 172.21.15.98:0/489279410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81029e0 0x7f41d8102e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009860 tx=0x7f41c402bb70 comp rx=0 tx=0).stop 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 -- 172.21.15.98:0/489279410 shutdown_connections 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 --2- 172.21.15.98:0/489279410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81033a0 0x7f41d8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 --2- 172.21.15.98:0/489279410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81029e0 0x7f41d8102e60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 --2- 172.21.15.98:0/489279410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81017f0 0x7f41d8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 -- 172.21.15.98:0/489279410 >> 172.21.15.98:0/489279410 conn(0x7f41d80fcd80 msgr2=0x7f41d80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 -- 172.21.15.98:0/489279410 shutdown_connections 2024-03-16T12:21:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.887+0000 7f41dd800640 1 -- 172.21.15.98:0/489279410 wait complete. 2024-03-16T12:21:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 Processor -- start 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 -- start start 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81017f0 0x7f41d8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81033a0 0x7f41d8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41d810d470 con 0x7f41d81033a0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41d810d2f0 con 0x7f41d81029e0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.888+0000 7f41dd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41d810d5f0 con 0x7f41d81017f0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81017f0 0x7f41d8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51146/0 (socket says 172.21.15.98:51146) 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 -- 172.21.15.98:0/888148931 learned_addr learned my addr 172.21.15.98:0/888148931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d7400640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81033a0 0x7f41d8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 -- 172.21.15.98:0/888148931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81017f0 msgr2=0x7f41d8075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81017f0 0x7f41d8075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 -- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81033a0 msgr2=0x7f41d8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81033a0 0x7f41d8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.889+0000 7f41d6000640 1 -- 172.21.15.98:0/888148931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d80749a0 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41d6000640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41c40349e0 tx=0x7f41c4032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4041030 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d81a1ba0 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c402c510 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4039040 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.890+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d81a2140 con 0x7f41d81029e0 2024-03-16T12:21:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.891+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41d8108b10 con 0x7f41d81029e0 2024-03-16T12:21:21.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.893+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f41c4044070 con 0x7f41d81029e0 2024-03-16T12:21:21.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.895+0000 7f41cf400640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 0x7f41bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:21.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.895+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f41c40f0330 con 0x7f41d81029e0 2024-03-16T12:21:21.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.895+0000 7f41d6a00640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 0x7f41bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:21.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.896+0000 7f41d6a00640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 0x7f41bc0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009530 tx=0x7f41c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:21.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:21.897+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41c40bd010 con 0x7f41d81029e0 2024-03-16T12:21:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.113+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41d802d330 con 0x7f41d81029e0 2024-03-16T12:21:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.115+0000 7f41cf400640 1 -- 172.21.15.98:0/888148931 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f41c40bc910 con 0x7f41d81029e0 2024-03-16T12:21:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:22.097 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 msgr2=0x7f41bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 0x7f41bc0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009530 tx=0x7f41c8009260 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 msgr2=0x7f41d8073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41c40349e0 tx=0x7f41c4032450 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 shutdown_connections 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f41bc0a2800 0x7f41bc0a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d81033a0 0x7f41d8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41d81029e0 0x7f41d8073bd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 --2- 172.21.15.98:0/888148931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41d81017f0 0x7f41d8075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 >> 172.21.15.98:0/888148931 conn(0x7f41d80fcd80 msgr2=0x7f41d8105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.117+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 shutdown_connections 2024-03-16T12:21:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.118+0000 7f41dd800640 1 -- 172.21.15.98:0/888148931 wait complete. 2024-03-16T12:21:22.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:21:22.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:21:22.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3793455698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:22 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3022412735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:22.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:22.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3793455698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:22 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3022412735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.319+0000 7fc9c5800640 1 -- 172.21.15.98:0/949619946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0109140 msgr2=0x7fc9c010fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.319+0000 7fc9c5800640 1 --2- 172.21.15.98:0/949619946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0109140 0x7fc9c010fb20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009900 tx=0x7fc9a802bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 -- 172.21.15.98:0/949619946 shutdown_connections 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 --2- 172.21.15.98:0/949619946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c0110060 0x7fc9c0112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 --2- 172.21.15.98:0/949619946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0109140 0x7fc9c010fb20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 --2- 172.21.15.98:0/949619946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c01087e0 0x7fc9c0108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 -- 172.21.15.98:0/949619946 >> 172.21.15.98:0/949619946 conn(0x7fc9c006e9e0 msgr2=0x7fc9c0070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 -- 172.21.15.98:0/949619946 shutdown_connections 2024-03-16T12:21:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.320+0000 7fc9c5800640 1 -- 172.21.15.98:0/949619946 wait complete. 2024-03-16T12:21:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 Processor -- start 2024-03-16T12:21:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 -- start start 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 0x7fc9c010b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 0x7fc9c0109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c0115c80 con 0x7fc9c01087e0 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c0115b00 con 0x7fc9c0110060 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.321+0000 7fc9c5800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c0115e00 con 0x7fc9c0109140 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9bea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 0x7fc9c010b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48692/0 (socket says 172.21.15.98:48692) 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 -- 172.21.15.98:0/1512057918 learned_addr learned my addr 172.21.15.98:0/1512057918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9bea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 0x7fc9c010b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35598/0 (socket says 172.21.15.98:35598) 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9bf400640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 0x7fc9c0109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 -- 172.21.15.98:0/1512057918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 msgr2=0x7fc9c0109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 0x7fc9c0109ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 -- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 msgr2=0x7fc9c010b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 0x7fc9c010b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 -- 172.21.15.98:0/1512057918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c010a880 con 0x7fc9c0109140 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9bf400640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 0x7fc9c0109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b6000640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009230 tx=0x7fc9a8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.322+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8041070 con 0x7fc9c0109140 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.323+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9a80044d0 con 0x7fc9c0109140 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.323+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8034440 con 0x7fc9c0109140 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.323+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c01aa440 con 0x7fc9c0109140 2024-03-16T12:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.323+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c01aa9e0 con 0x7fc9c0109140 2024-03-16T12:21:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.324+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c010e930 con 0x7fc9c0109140 2024-03-16T12:21:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.326+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc9a8044070 con 0x7fc9c0109140 2024-03-16T12:21:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.327+0000 7fc9b7e00640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 0x7fc9a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.327+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc9a80f0d80 con 0x7fc9c0109140 2024-03-16T12:21:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.327+0000 7fc9bea00640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 0x7fc9a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.328+0000 7fc9bea00640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 0x7fc9a40a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009bd0 tx=0x7fc9b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.334+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc9a80bda60 con 0x7fc9c0109140 2024-03-16T12:21:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: pgmap v400: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3793455698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:22.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:22 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3022412735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.534+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9c002d340 con 0x7fc9c0109140 2024-03-16T12:21:22.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.535+0000 7fc9b7e00640 1 -- 172.21.15.98:0/1512057918 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fc9a80bd360 con 0x7fc9c0109140 2024-03-16T12:21:22.517 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:21:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 msgr2=0x7fc9a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 0x7fc9a40a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009bd0 tx=0x7fc9b0005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 msgr2=0x7fc9c0109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009230 tx=0x7fc9a8005d50 comp rx=0 tx=0).stop 2024-03-16T12:21:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 shutdown_connections 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc9a40a2880 0x7fc9a40a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc9c0110060 0x7fc9c0109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9c0109140 0x7fc9c0109ab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 --2- 172.21.15.98:0/1512057918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9c01087e0 0x7fc9c010b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 >> 172.21.15.98:0/1512057918 conn(0x7fc9c006e9e0 msgr2=0x7fc9c010bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 shutdown_connections 2024-03-16T12:21:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.538+0000 7fc9c5800640 1 -- 172.21.15.98:0/1512057918 wait complete. 2024-03-16T12:21:22.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.741+0000 7f0971400640 1 -- 172.21.15.98:0/830502990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 msgr2=0x7f096c10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.741+0000 7f0971400640 1 --2- 172.21.15.98:0/830502990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c10be40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0964009be0 tx=0x7f096402bef0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 -- 172.21.15.98:0/830502990 shutdown_connections 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 --2- 172.21.15.98:0/830502990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c10be40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 --2- 172.21.15.98:0/830502990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 0x7f096c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 --2- 172.21.15.98:0/830502990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 -- 172.21.15.98:0/830502990 >> 172.21.15.98:0/830502990 conn(0x7f096c0fef90 msgr2=0x7f096c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 -- 172.21.15.98:0/830502990 shutdown_connections 2024-03-16T12:21:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.742+0000 7f0971400640 1 -- 172.21.15.98:0/830502990 wait complete. 2024-03-16T12:21:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 Processor -- start 2024-03-16T12:21:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 -- start start 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 0x7f096c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f096c10f6d0 con 0x7f096c1055b0 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f096c10f550 con 0x7f096c104bf0 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0971400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f096c10f850 con 0x7f096c103a00 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f096a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f096b200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f096a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48716/0 (socket says 172.21.15.98:48716) 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f096a800640 1 -- 172.21.15.98:0/3264216541 learned_addr learned my addr 172.21.15.98:0/3264216541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.743+0000 7f0969e00640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 0x7f096c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 -- 172.21.15.98:0/3264216541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 msgr2=0x7f096c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 0x7f096c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 -- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 msgr2=0x7f096c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 -- 172.21.15.98:0/3264216541 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c1a18f0 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096b200640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f096a800640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f095400e9c0 tx=0x7f095400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095400cd80 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096c1a1be0 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09540035f0 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954010690 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.744+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096c1a20e0 con 0x7f096c103a00 2024-03-16T12:21:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.745+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f096c10acc0 con 0x7f096c103a00 2024-03-16T12:21:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.748+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f09540037a0 con 0x7f096c103a00 2024-03-16T12:21:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.749+0000 7f0963400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 0x7f09500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.749+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f09540cf190 con 0x7f096c103a00 2024-03-16T12:21:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.749+0000 7f0969e00640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 0x7f09500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.750+0000 7f0969e00640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 0x7f09500a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f096c19b1b0 tx=0x7f095c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.753+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f095409be70 con 0x7f096c103a00 2024-03-16T12:21:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.960+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f096c0623e0 con 0x7f096c103a00 2024-03-16T12:21:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.961+0000 7f0963400640 1 -- 172.21.15.98:0/3264216541 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f095409b770 con 0x7f096c103a00 2024-03-16T12:21:22.942 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 msgr2=0x7f09500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 0x7f09500a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f096c19b1b0 tx=0x7f095c009260 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 msgr2=0x7f096c199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f095400e9c0 tx=0x7f095400ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 shutdown_connections 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f09500a2800 0x7f09500a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c1055b0 0x7f096c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f096c104bf0 0x7f096c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 --2- 172.21.15.98:0/3264216541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f096c103a00 0x7f096c199cb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 >> 172.21.15.98:0/3264216541 conn(0x7f096c0fef90 msgr2=0x7f096c107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:22.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 shutdown_connections 2024-03-16T12:21:22.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:22.963+0000 7f0971400640 1 -- 172.21.15.98:0/3264216541 wait complete. 2024-03-16T12:21:22.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:21:23.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 -- 172.21.15.98:0/412816631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6624075820 msgr2=0x7f6624073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 --2- 172.21.15.98:0/412816631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6624075820 0x7f6624073c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6618009900 tx=0x7f661802bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 -- 172.21.15.98:0/412816631 shutdown_connections 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 --2- 172.21.15.98:0/412816631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f662410ddc0 0x7f66241101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 --2- 172.21.15.98:0/412816631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624074240 0x7f66240746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 --2- 172.21.15.98:0/412816631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6624075820 0x7f6624073c70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 -- 172.21.15.98:0/412816631 >> 172.21.15.98:0/412816631 conn(0x7f66240fcd80 msgr2=0x7f66240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 -- 172.21.15.98:0/412816631 shutdown_connections 2024-03-16T12:21:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.177+0000 7f662bc00640 1 -- 172.21.15.98:0/412816631 wait complete. 2024-03-16T12:21:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 Processor -- start 2024-03-16T12:21:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 -- start start 2024-03-16T12:21:23.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624075820 0x7f662419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f662410ddc0 0x7f66241a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6624113a90 con 0x7f6624075820 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6624113910 con 0x7f6624074240 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.178+0000 7f662bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6624113c10 con 0x7f662410ddc0 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51198/0 (socket says 172.21.15.98:51198) 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 -- 172.21.15.98:0/2566735264 learned_addr learned my addr 172.21.15.98:0/2566735264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6623e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624075820 0x7f662419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629c00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f662410ddc0 0x7f66241a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 -- 172.21.15.98:0/2566735264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f662410ddc0 msgr2=0x7f66241a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f662410ddc0 0x7f66241a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 -- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624075820 msgr2=0x7f662419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624075820 0x7f662419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 -- 172.21.15.98:0/2566735264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66241a5d40 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.179+0000 7f6629200640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6618005ab0 tx=0x7f661802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.180+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618041070 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.180+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6618004450 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.180+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66241a5fd0 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.180+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618003d60 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.180+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66241a6430 con 0x7f6624074240 2024-03-16T12:21:23.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.181+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6624103f90 con 0x7f6624074240 2024-03-16T12:21:23.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.183+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6618044070 con 0x7f6624074240 2024-03-16T12:21:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.184+0000 7f6621600640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 0x7f660c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.184+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f66180f0f90 con 0x7f6624074240 2024-03-16T12:21:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.185+0000 7f6623e00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 0x7f660c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.185+0000 7f6623e00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 0x7f660c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f66100095f0 tx=0x7f6610005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.186+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66180bd620 con 0x7f6624074240 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/888148931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1512057918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:23 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3264216541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/888148931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1512057918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:23 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3264216541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/888148931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1512057918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:23.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:23 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3264216541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.394+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f662402d400 con 0x7f6624074240 2024-03-16T12:21:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.395+0000 7f6621600640 1 -- 172.21.15.98:0/2566735264 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f66180c16c0 con 0x7f6624074240 2024-03-16T12:21:23.377 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:21:23.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 msgr2=0x7f660c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 0x7f660c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f66100095f0 tx=0x7f6610005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:23.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 msgr2=0x7f662419e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6618005ab0 tx=0x7f661802c6c0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 shutdown_connections 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f660c0a2880 0x7f660c0a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f662410ddc0 0x7f66241a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6624075820 0x7f662419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 --2- 172.21.15.98:0/2566735264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6624074240 0x7f662419e120 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 >> 172.21.15.98:0/2566735264 conn(0x7f66240fcd80 msgr2=0x7f6624101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 shutdown_connections 2024-03-16T12:21:23.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.398+0000 7f662bc00640 1 -- 172.21.15.98:0/2566735264 wait complete. 2024-03-16T12:21:23.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1912138629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f181055d0 msgr2=0x7f4f1810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1912138629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f181055d0 0x7f4f1810be60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009ca0 tx=0x7f4f1002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1912138629 shutdown_connections 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1912138629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f181055d0 0x7f4f1810be60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1912138629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 0x7f4f18105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1912138629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f18103a20 0x7f4f18103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1912138629 >> 172.21.15.98:0/1912138629 conn(0x7f4f180fef90 msgr2=0x7f4f181013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1912138629 shutdown_connections 2024-03-16T12:21:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.610+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1912138629 wait complete. 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.611+0000 7f4f1ce00640 1 Processor -- start 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.611+0000 7f4f1ce00640 1 -- start start 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 0x7f4f1819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 0x7f4f181a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810f6a0 con 0x7f4f18104c10 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810f520 con 0x7f4f181055d0 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f1ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1810f820 con 0x7f4f18103a20 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48742/0 (socket says 172.21.15.98:48742) 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 -- 172.21.15.98:0/1434370854 learned_addr learned my addr 172.21.15.98:0/1434370854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15400640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 0x7f4f1819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f16800640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 0x7f4f181a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 -- 172.21.15.98:0/1434370854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 msgr2=0x7f4f181a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 0x7f4f181a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 -- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 msgr2=0x7f4f1819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 0x7f4f1819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f15e00640 1 -- 172.21.15.98:0/1434370854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f181a18f0 con 0x7f4f18103a20 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.612+0000 7f4f16800640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 0x7f4f181a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f15e00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800e9c0 tx=0x7f4f0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0800cd80 con 0x7f4f18103a20 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f080035f0 con 0x7f4f18103a20 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f181a1be0 con 0x7f4f18103a20 2024-03-16T12:21:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f08010690 con 0x7f4f18103a20 2024-03-16T12:21:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.613+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f181a2130 con 0x7f4f18103a20 2024-03-16T12:21:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.614+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f1810ad40 con 0x7f4f18103a20 2024-03-16T12:21:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.616+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4f080037a0 con 0x7f4f18103a20 2024-03-16T12:21:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.617+0000 7f4f0ea00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 0x7f4efc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.617+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4f080cf0f0 con 0x7f4f18103a20 2024-03-16T12:21:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.618+0000 7f4f15400640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 0x7f4efc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.618+0000 7f4f15400640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 0x7f4efc0a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f1819b1b0 tx=0x7f4f04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.621+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f0809bdd0 con 0x7f4f18103a20 2024-03-16T12:21:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.829+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f180623e0 con 0x7f4f18103a20 2024-03-16T12:21:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.830+0000 7f4f0ea00640 1 -- 172.21.15.98:0/1434370854 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f4f0809b6d0 con 0x7f4f18103a20 2024-03-16T12:21:23.812 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 msgr2=0x7f4efc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 0x7f4efc0a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f1819b1b0 tx=0x7f4f04005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 msgr2=0x7f4f18199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800e9c0 tx=0x7f4f0800ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 shutdown_connections 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f4efc0a2880 0x7f4efc0a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f181055d0 0x7f4f181a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f18104c10 0x7f4f1819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 --2- 172.21.15.98:0/1434370854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f18103a20 0x7f4f18199cb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.832+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 >> 172.21.15.98:0/1434370854 conn(0x7f4f180fef90 msgr2=0x7f4f18107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.833+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 shutdown_connections 2024-03-16T12:21:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:23.833+0000 7f4f1ce00640 1 -- 172.21.15.98:0/1434370854 wait complete. 2024-03-16T12:21:23.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:21:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 -- 172.21.15.98:0/1516256926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0103a00 msgr2=0x7fe1e0103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 --2- 172.21.15.98:0/1516256926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0103a00 0x7fe1e0103e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009900 tx=0x7fe1d402bc50 comp rx=0 tx=0).stop 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 -- 172.21.15.98:0/1516256926 shutdown_connections 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 --2- 172.21.15.98:0/1516256926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 0x7fe1e010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 --2- 172.21.15.98:0/1516256926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0104bf0 0x7fe1e0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 --2- 172.21.15.98:0/1516256926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0103a00 0x7fe1e0103e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 -- 172.21.15.98:0/1516256926 >> 172.21.15.98:0/1516256926 conn(0x7fe1e00fef90 msgr2=0x7fe1e01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 -- 172.21.15.98:0/1516256926 shutdown_connections 2024-03-16T12:21:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.042+0000 7fe1e7800640 1 -- 172.21.15.98:0/1516256926 wait complete. 2024-03-16T12:21:24.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.043+0000 7fe1e7800640 1 Processor -- start 2024-03-16T12:21:24.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.043+0000 7fe1e7800640 1 -- start start 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0103a00 0x7fe1e0199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 0x7fe1e01a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e010f6d0 con 0x7fe1e0103a00 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e010f550 con 0x7fe1e01055b0 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e7800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e010f850 con 0x7fe1e0104bf0 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e5800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 0x7fe1e01a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48758/0 (socket says 172.21.15.98:48758) 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 -- 172.21.15.98:0/3268912810 learned_addr learned my addr 172.21.15.98:0/3268912810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1e4e00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0103a00 0x7fe1e0199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 -- 172.21.15.98:0/3268912810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 msgr2=0x7fe1e01a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 0x7fe1e01a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 -- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0103a00 msgr2=0x7fe1e0199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0103a00 0x7fe1e0199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.044+0000 7fe1dfe00640 1 -- 172.21.15.98:0/3268912810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e01a1910 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1dfe00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00d950 tx=0x7fe1cc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc017070 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1cc009c30 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc00f6b0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e01a1c00 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.045+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e01a20d0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.046+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e010acc0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.049+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe1cc0109d0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.051+0000 7fe1dd600640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 0x7fe1c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.051+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe1cc0ce990 con 0x7fe1e0104bf0 2024-03-16T12:21:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.052+0000 7fe1e4e00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 0x7fe1c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.052+0000 7fe1e4e00640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 0x7fe1c80a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402c190 tx=0x7fe1d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.055+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1cc009de0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:24 smithi098 ceph-mon[31633]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:24.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:24 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2566735264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:24 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:24 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1434370854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:24 smithi153 ceph-mon[36901]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:24 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2566735264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:24 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:24 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1434370854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.295+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1e002d1b0 con 0x7fe1e0104bf0 2024-03-16T12:21:24.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.296+0000 7fe1dd600640 1 -- 172.21.15.98:0/3268912810 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7fe1cc09b080 con 0x7fe1e0104bf0 2024-03-16T12:21:24.277 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:24.277 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:19:07.477601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24553,24599,24542]},"id":1}]} 2024-03-16T12:21:24.278 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 msgr2=0x7fe1c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 0x7fe1c80a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402c190 tx=0x7fe1d4000bc0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 msgr2=0x7fe1e019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00d950 tx=0x7fe1cc00de10 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 shutdown_connections 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe1c80a2880 0x7fe1c80a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e01055b0 0x7fe1e01a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1e0104bf0 0x7fe1e019a240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 --2- 172.21.15.98:0/3268912810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e0103a00 0x7fe1e0199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.298+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 >> 172.21.15.98:0/3268912810 conn(0x7fe1e00fef90 msgr2=0x7fe1e0107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.299+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 shutdown_connections 2024-03-16T12:21:24.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.299+0000 7fe1e7800640 1 -- 172.21.15.98:0/3268912810 wait complete. 2024-03-16T12:21:24.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:24.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:24 smithi173 ceph-mon[36632]: pgmap v401: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:24.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:24 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2566735264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:24 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:24.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:24 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1434370854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 -- 172.21.15.98:0/174147929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda4100a50 msgr2=0x7fdda4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 --2- 172.21.15.98:0/174147929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda4100a50 0x7fdda4100eb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009900 tx=0x7fdd9002bcd0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 -- 172.21.15.98:0/174147929 shutdown_connections 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 --2- 172.21.15.98:0/174147929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda41013f0 0x7fdda4101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 --2- 172.21.15.98:0/174147929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda4100a50 0x7fdda4100eb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 --2- 172.21.15.98:0/174147929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 0x7fdda4100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 -- 172.21.15.98:0/174147929 >> 172.21.15.98:0/174147929 conn(0x7fdda40fbc70 msgr2=0x7fdda40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 -- 172.21.15.98:0/174147929 shutdown_connections 2024-03-16T12:21:24.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.517+0000 7fddab600640 1 -- 172.21.15.98:0/174147929 wait complete. 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.518+0000 7fddab600640 1 Processor -- start 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 -- start start 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 0x7fdda4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda41013f0 0x7fdda41a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410f6b0 con 0x7fdda41000f0 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410f530 con 0x7fdda41013f0 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fddab600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410f830 con 0x7fdda4100a50 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fdda3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fdda8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 0x7fdda4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fdda3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48780/0 (socket says 172.21.15.98:48780) 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fdda3e00640 1 -- 172.21.15.98:0/383913354 learned_addr learned my addr 172.21.15.98:0/383913354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.519+0000 7fdda9600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda41013f0 0x7fdda41a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 -- 172.21.15.98:0/383913354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda41013f0 msgr2=0x7fdda41a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda41013f0 0x7fdda41a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 -- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 msgr2=0x7fdda4199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 0x7fdda4199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 -- 172.21.15.98:0/383913354 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda41a1920 con 0x7fdda4100a50 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda3e00640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd90000c00 tx=0x7fdd9002c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90043020 con 0x7fdda4100a50 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd900054e0 con 0x7fdda4100a50 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda41a1bb0 con 0x7fdda4100a50 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90003b20 con 0x7fdda4100a50 2024-03-16T12:21:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.520+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda41a2100 con 0x7fdda4100a50 2024-03-16T12:21:24.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.521+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda410afd0 con 0x7fdda4100a50 2024-03-16T12:21:24.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.523+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdd90046020 con 0x7fdda4100a50 2024-03-16T12:21:24.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.525+0000 7fdda1600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 0x7fdd8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.525+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdd900f0e80 con 0x7fdda4100a50 2024-03-16T12:21:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.525+0000 7fdda8c00640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 0x7fdd8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.526+0000 7fdda8c00640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 0x7fdd8c0a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdda419b1e0 tx=0x7fdd98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.528+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd9004d020 con 0x7fdda4100a50 2024-03-16T12:21:24.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.738+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdda40623e0 con 0x7fdda4100a50 2024-03-16T12:21:24.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.740+0000 7fdda1600640 1 -- 172.21.15.98:0/383913354 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7fdd900bdb60 con 0x7fdda4100a50 2024-03-16T12:21:24.721 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:24.722 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 msgr2=0x7fdd8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 0x7fdd8c0a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdda419b1e0 tx=0x7fdd98005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 msgr2=0x7fdda419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd90000c00 tx=0x7fdd9002c6f0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 shutdown_connections 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdd8c0a2800 0x7fdd8c0a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdda41013f0 0x7fdda41a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4100a50 0x7fdda419a250 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 --2- 172.21.15.98:0/383913354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdda41000f0 0x7fdda4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 >> 172.21.15.98:0/383913354 conn(0x7fdda40fbc70 msgr2=0x7fdda40fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 shutdown_connections 2024-03-16T12:21:24.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.742+0000 7fddab600640 1 -- 172.21.15.98:0/383913354 wait complete. 2024-03-16T12:21:24.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T12:18:40.454968+0000', 'modified': '2024-03-16T12:19:07.477601+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24599, 'mds_1': 24553, 'mds_2': 24542}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6835/4013234440', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 4013234440}, {'type': 'v1', 'addr': '172.21.15.153:6835', 'nonce': 4013234440}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2799668647', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2799668647}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2799668647}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/4083501894', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 4083501894}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 4083501894}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.153:6837/419588181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 419588181}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 419588181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.173:6839/205594788', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 205594788}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 205594788}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/542416413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 542416413}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 542416413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24553, 24599, 24542]} 2024-03-16T12:21:24.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T12:21:24.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 -- 172.21.15.98:0/4273403919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81028d0 msgr2=0x7f19e8102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 --2- 172.21.15.98:0/4273403919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81028d0 0x7f19e8102d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009860 tx=0x7f19d402bb70 comp rx=0 tx=0).stop 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 -- 172.21.15.98:0/4273403919 shutdown_connections 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 --2- 172.21.15.98:0/4273403919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e8103290 0x7f19e8109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 --2- 172.21.15.98:0/4273403919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81028d0 0x7f19e8102d50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 --2- 172.21.15.98:0/4273403919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e81016e0 0x7f19e8101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 -- 172.21.15.98:0/4273403919 >> 172.21.15.98:0/4273403919 conn(0x7f19e80fcc50 msgr2=0x7f19e80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 -- 172.21.15.98:0/4273403919 shutdown_connections 2024-03-16T12:21:24.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.950+0000 7f19ef000640 1 -- 172.21.15.98:0/4273403919 wait complete. 2024-03-16T12:21:24.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.951+0000 7f19ef000640 1 Processor -- start 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.951+0000 7f19ef000640 1 -- start start 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 0x7f19e8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 0x7f19e819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d360 con 0x7f19e8103290 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d1e0 con 0x7f19e81028d0 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ef000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d4e0 con 0x7f19e81016e0 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19edc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 0x7f19e8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19edc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48784/0 (socket says 172.21.15.98:48784) 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19edc00640 1 -- 172.21.15.98:0/3857472098 learned_addr learned my addr 172.21.15.98:0/3857472098 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ee600640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 0x7f19e819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 -- 172.21.15.98:0/3857472098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 msgr2=0x7f19e81979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 -- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 msgr2=0x7f19e819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 0x7f19e819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ed200640 1 -- 172.21.15.98:0/3857472098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e819f600 con 0x7f19e81028d0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19edc00640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.952+0000 7f19ee600640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 0x7f19e819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19ed200640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 0x7f19e8197f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19d402c0b0 tx=0x7f19d4032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19e6000640 1 -- 172.21.15.98:0/3857472098 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4041020 con 0x7f19e81028d0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19e6000640 1 -- 172.21.15.98:0/3857472098 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19d402c9f0 con 0x7f19e81028d0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19e6000640 1 -- 172.21.15.98:0/3857472098 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4039040 con 0x7f19e81028d0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e819f890 con 0x7f19e81028d0 2024-03-16T12:21:24.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.953+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e819fd00 con 0x7f19e81028d0 2024-03-16T12:21:24.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 Processor -- start 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19e6000640 1 -- 172.21.15.98:0/3857472098 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f19d4044020 con 0x7f19e81028d0 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 -- start start 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478280cb0 0x5624782837a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478283e10 0x562478290c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5624781e4c50 con 0x562478280cb0 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56247818d9d0 con 0x562478283e10 2024-03-16T12:21:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.956+0000 7f19f33ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624781e0cf0 con 0x5624782802c0 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48794/0 (socket says 172.21.15.98:48794) 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478283e10 0x562478290c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 -- 172.21.15.98:0/1460123119 learned_addr learned my addr 172.21.15.98:0/1460123119 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19edc00640 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478280cb0 0x5624782837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 -- 172.21.15.98:0/1460123119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478283e10 msgr2=0x562478290c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478283e10 0x562478290c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 -- 172.21.15.98:0/1460123119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478280cb0 msgr2=0x5624782837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478280cb0 0x5624782837a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 -- 172.21.15.98:0/1460123119 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624781a7f10 con 0x5624782802c0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.957+0000 7f19ee600640 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009cc0 tx=0x7f19e00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b46ef5536f9f6ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19daa00640 1 -- 172.21.15.98:0/1460123119 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0012450 con 0x5624782802c0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19daa00640 1 -- 172.21.15.98:0/1460123119 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e0012ad0 con 0x5624782802c0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19daa00640 1 -- 172.21.15.98:0/1460123119 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e000fc30 con 0x5624782802c0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19e6000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 0x7f19d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19e6000640 1 -- 172.21.15.98:0/3857472098 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19d40f06b0 con 0x7f19e81028d0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19edc00640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 0x7f19d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19f33ac740 1 -- 172.21.15.98:0/1460123119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 msgr2=0x5624782806e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.958+0000 7f19f33ac740 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009cc0 tx=0x7f19e00117b0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 -- 172.21.15.98:0/1460123119 shutdown_connections 2024-03-16T12:21:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478283e10 0x562478290c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478280cb0 0x5624782837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 --2- 172.21.15.98:0/1460123119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624782802c0 0x5624782806e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 -- 172.21.15.98:0/1460123119 >> 172.21.15.98:0/1460123119 conn(0x56247827b8b0 msgr2=0x56247827bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 -- 172.21.15.98:0/1460123119 shutdown_connections 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 -- 172.21.15.98:0/1460123119 wait complete. 2024-03-16T12:21:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19edc00640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 0x7f19d00a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x56247827c820 tx=0x7f19dc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.959+0000 7f19f33ac740 1 -- start start 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 Processor -- start 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562478291680 con 0x5624782802c0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5624781a0e20 con 0x562478280cb0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562478291800 con 0x562478283e10 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19f33ac740 10 client.0 _ll_register_callbacks cb 0x5624781a07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19edc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51274/0 (socket says 172.21.15.98:51274) 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19edc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35718/0 (socket says 172.21.15.98:35718) 2024-03-16T12:21:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48796/0 (socket says 172.21.15.98:48796) 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.960+0000 7f19ed200640 1 -- 172.21.15.98:0/3193683116 learned_addr learned my addr 172.21.15.98:0/3193683116 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 -- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 msgr2=0x5624782931e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 -- 172.21.15.98:0/3193683116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 msgr2=0x562478292ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 -- 172.21.15.98:0/3193683116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562477ef92d0 con 0x562478283e10 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ed200640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19edc00640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19ee600640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f19e0011cb0 tx=0x7f19e0005cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2b1ea09f23ad86 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19cf400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0029020 con 0x562478283e10 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19e0012ad0 con 0x562478283e10 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19e000fbc0 con 0x562478283e10 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.961+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f19e0034050 con 0x562478283e10 2024-03-16T12:21:24.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 10 client.25336 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562477fc36b0 con 0x562478283e10 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e0012ad0 con 0x562478283e10 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f19e00046c0 con 0x562478283e10 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 10 client.25336 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 10 client.25336 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 20 client.25336 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x56247829f6d0 0x5624782a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 4 client.25336 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624782422b0 con 0x56247829f6d0 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.962+0000 7f19edc00640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x56247829f6d0 0x5624782a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.963+0000 7f19edc00640 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x56247829f6d0 0x5624782a0b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8fab16dc148729 server_cookie=561c60396974eaf in_seq=0 out_seq=0 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.963+0000 7f19cf400640 10 client.25336 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19cf400640 1 -- 172.21.15.98:0/3193683116 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624782422b0 con 0x56247829f6d0 2024-03-16T12:21:24.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19cf400640 10 client.25336 handle_command_reply: tid=0 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 client.25336 shutdown 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 msgr2=0x56247829a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f19e0011cb0 tx=0x7f19e0005cd0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 shutdown_connections 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x56247829f6d0 0x5624782a0b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247829a220 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x5624782931e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3193683116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x562478292ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19cf400640 0 client.25336 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 >> 172.21.15.98:0/3193683116 conn(0x56247827b8b0 msgr2=0x56247827aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 shutdown_connections 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 1 -- 172.21.15.98:0/3193683116 wait complete. 2024-03-16T12:21:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.964+0000 7f19f33ac740 20 client.25336 trim_cache size 0 max 16384 2024-03-16T12:21:24.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 Processor -- start 2024-03-16T12:21:24.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 -- start start 2024-03-16T12:21:24.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x56247827bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247827a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56247829ee80 con 0x5624782802c0 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56247829f550 con 0x562478280cb0 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.983+0000 7f19f33ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56247829fc20 con 0x562478283e10 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19edc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x56247827bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247827a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51288/0 (socket says 172.21.15.98:51288) 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 -- 172.21.15.98:0/44600589 learned_addr learned my addr 172.21.15.98:0/44600589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 -- 172.21.15.98:0/44600589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 msgr2=0x56247827a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247827a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 -- 172.21.15.98:0/44600589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 msgr2=0x56247827bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x56247827bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 -- 172.21.15.98:0/44600589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247827ff00 con 0x562478280cb0 2024-03-16T12:21:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ee600640 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247827a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:24.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19ed200640 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f19d40f0950 tx=0x7f19d4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70fc3ca73bf4b969 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.984+0000 7f19da000640 1 -- 172.21.15.98:0/44600589 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d40b8f30 con 0x562478280cb0 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19da000640 1 -- 172.21.15.98:0/44600589 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19d4032e50 con 0x562478280cb0 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19da000640 1 -- 172.21.15.98:0/44600589 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d40bc5f0 con 0x562478280cb0 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 -- 172.21.15.98:0/44600589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 msgr2=0x56247829a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f19d40f0950 tx=0x7f19d4005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 -- 172.21.15.98:0/44600589 shutdown_connections 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478283e10 0x56247827a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562478280cb0 0x56247829a140 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 --2- 172.21.15.98:0/44600589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624782802c0 0x56247827bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 -- 172.21.15.98:0/44600589 >> 172.21.15.98:0/44600589 conn(0x56247827b8b0 msgr2=0x562478292c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 -- 172.21.15.98:0/44600589 shutdown_connections 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.985+0000 7f19f33ac740 1 -- 172.21.15.98:0/44600589 wait complete. 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 -- start start 2024-03-16T12:21:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 Processor -- start 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 0x562478295b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478283e10 0x562478298af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5624782a09c0 con 0x562478283e10 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5624782a02f0 con 0x5624782802c0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624782a1090 con 0x562478280cb0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19f33ac740 10 client.0 _ll_register_callbacks cb 0x7f19e0009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.986+0000 7f19edc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51292/0 (socket says 172.21.15.98:51292) 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 -- 172.21.15.98:0/3957723780 learned_addr learned my addr 172.21.15.98:0/3957723780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19ed200640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 0x562478295b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19ee600640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478283e10 0x562478298af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 -- 172.21.15.98:0/3957723780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 msgr2=0x562478295b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 0x562478295b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 -- 172.21.15.98:0/3957723780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478283e10 msgr2=0x562478298af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478283e10 0x562478298af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 -- 172.21.15.98:0/3957723780 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624782990e0 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19ed200640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 0x562478295b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19edc00640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009330 tx=0x7f19dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d793ebe302eb9c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19cf400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0197f0 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19dc019990 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.987+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d4032e50 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f19dc019b90 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 10 client.25352 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56247829a450 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19d4032e50 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f19dc00b040 con 0x5624782802c0 2024-03-16T12:21:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 10 client.25352 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 10 client.25352 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 20 client.25352 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x562478309e50 0x562478308b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 4 client.25352 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.988+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "status"}) -- 0x5624781f2190 con 0x562478309e50 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19ed200640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x562478309e50 0x562478308b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19ed200640 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x562478309e50 0x562478308b60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f6df59f5a7b0717 server_cookie=99d5725a3139c700 in_seq=0 out_seq=0 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19cf400640 10 client.25352 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19cf400640 1 -- 172.21.15.98:0/3957723780 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5624781f2190 con 0x562478309e50 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19cf400640 10 client.25352 handle_command_reply: tid=0 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 client.25352 shutdown 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 msgr2=0x5624782955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009330 tx=0x7f19dc004a40 comp rx=0 tx=0).stop 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 shutdown_connections 2024-03-16T12:21:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x562478309e50 0x562478308b60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562478283e10 0x562478298af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562478280cb0 0x562478295b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 --2- 172.21.15.98:0/3957723780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5624782802c0 0x5624782955e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19cf400640 0 client.25352 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.989+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 >> 172.21.15.98:0/3957723780 conn(0x56247827b8b0 msgr2=0x56247827a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 shutdown_connections 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19f33ac740 1 -- 172.21.15.98:0/3957723780 wait complete. 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19f33ac740 20 client.25352 trim_cache size 0 max 16384 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:21:24.976 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24542, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 138.07031715400001, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 184.667994128 2024-03-16T12:21:24.977 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 msgr2=0x7f19d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 0x7f19d00a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x56247827c820 tx=0x7f19dc0095c0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 msgr2=0x7f19e8197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 0x7f19e8197f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19d402c0b0 tx=0x7f19d4032430 comp rx=0 tx=0).stop 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 shutdown_connections 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19d00a2800 0x7f19d00a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19e8103290 0x7f19e819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19e81028d0 0x7f19e8197f20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.990+0000 7f19ef000640 1 --2- 172.21.15.98:0/3857472098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19e81016e0 0x7f19e81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.991+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 >> 172.21.15.98:0/3857472098 conn(0x7f19e80fcc50 msgr2=0x7f19e8105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.991+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 shutdown_connections 2024-03-16T12:21:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:24.991+0000 7f19ef000640 1 -- 172.21.15.98:0/3857472098 wait complete. 2024-03-16T12:21:24.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 -- 172.21.15.98:0/1759491158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c104ad0 msgr2=0x7f125c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 --2- 172.21.15.98:0/1759491158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c104ad0 0x7f125c104f50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1248009900 tx=0x7f124802bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 -- 172.21.15.98:0/1759491158 shutdown_connections 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 --2- 172.21.15.98:0/1759491158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 --2- 172.21.15.98:0/1759491158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c104ad0 0x7f125c104f50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 --2- 172.21.15.98:0/1759491158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c1038e0 0x7f125c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 -- 172.21.15.98:0/1759491158 >> 172.21.15.98:0/1759491158 conn(0x7f125c0fee50 msgr2=0x7f125c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 -- 172.21.15.98:0/1759491158 shutdown_connections 2024-03-16T12:21:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.189+0000 7f125b200640 1 -- 172.21.15.98:0/1759491158 wait complete. 2024-03-16T12:21:25.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.190+0000 7f125b200640 1 Processor -- start 2024-03-16T12:21:25.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.190+0000 7f125b200640 1 -- start start 2024-03-16T12:21:25.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c104ad0 0x7f125c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c1a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10f540 con 0x7f125c104ad0 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10f3c0 con 0x7f125c1038e0 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10f6c0 con 0x7f125c105490 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51314/0 (socket says 172.21.15.98:51314) 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 -- 172.21.15.98:0/3830786328 learned_addr learned my addr 172.21.15.98:0/3830786328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259400640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c104ad0 0x7f125c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125a800640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c1a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 -- 172.21.15.98:0/3830786328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 msgr2=0x7f125c1a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c1a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 -- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c104ad0 msgr2=0x7f125c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c104ad0 0x7f125c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f1259e00640 1 -- 172.21.15.98:0/3830786328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c1a17d0 con 0x7f125c1038e0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.191+0000 7f125a800640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f1259e00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f124c00e9c0 tx=0x7f124c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f1252a00640 1 -- 172.21.15.98:0/3830786328 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c00ccc0 con 0x7f125c1038e0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f1252a00640 1 -- 172.21.15.98:0/3830786328 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f124c0035f0 con 0x7f125c1038e0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125c1a1ac0 con 0x7f125c1038e0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f1252a00640 1 -- 172.21.15.98:0/3830786328 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c00ce60 con 0x7f125c1038e0 2024-03-16T12:21:25.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.192+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c1a1fe0 con 0x7f125c1038e0 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1263fe2740 1 Processor -- start 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1252a00640 1 -- 172.21.15.98:0/3830786328 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f124c0037a0 con 0x7f125c1038e0 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1263fe2740 1 -- start start 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fc5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.195+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc5ea0 0x557357fc8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f1263fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557357f24c50 con 0x557357fc0d30 2024-03-16T12:21:25.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f1263fe2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557357ecd9d0 con 0x557357fc5ea0 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f1263fe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557357f20cf0 con 0x557357fc0340 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f1259e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fc5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48842/0 (socket says 172.21.15.98:48842) 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 -- 172.21.15.98:0/3652760748 learned_addr learned my addr 172.21.15.98:0/3652760748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f1259400640 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc5ea0 0x557357fc8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 -- 172.21.15.98:0/3652760748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc5ea0 msgr2=0x557357fc8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc5ea0 0x557357fc8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 -- 172.21.15.98:0/3652760748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 msgr2=0x557357fc5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fc5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 -- 172.21.15.98:0/3652760748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557357ee7f10 con 0x557357fc0340 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.196+0000 7f125a800640 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1254011810 tx=0x7f1254011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb8f35121f90ee7e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1247400640 1 -- 172.21.15.98:0/3652760748 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254011310 con 0x557357fc0340 2024-03-16T12:21:25.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1247400640 1 -- 172.21.15.98:0/3652760748 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125400f9a0 con 0x557357fc0340 2024-03-16T12:21:25.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1247400640 1 -- 172.21.15.98:0/3652760748 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254005720 con 0x557357fc0340 2024-03-16T12:21:25.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1252a00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 0x7f12400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1252a00640 1 -- 172.21.15.98:0/3830786328 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f124c0cf180 con 0x7f125c1038e0 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1259e00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 0x7f12400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 -- 172.21.15.98:0/3652760748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 msgr2=0x557357fc0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1254011810 tx=0x7f1254011cd0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 -- 172.21.15.98:0/3652760748 shutdown_connections 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc5ea0 0x557357fc8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fc5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 --2- 172.21.15.98:0/3652760748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fc0760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 -- 172.21.15.98:0/3652760748 >> 172.21.15.98:0/3652760748 conn(0x557357fbb8b0 msgr2=0x557357fbbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 -- 172.21.15.98:0/3652760748 shutdown_connections 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.197+0000 7f1263fe2740 1 -- 172.21.15.98:0/3652760748 wait complete. 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.198+0000 7f1259e00640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 0x7f12400a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f124c0cc8c0 tx=0x7f124c0d2bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.198+0000 7f1263fe2740 1 -- start start 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.198+0000 7f1263fe2740 1 Processor -- start 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.198+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.198+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc5ea0 0x557357fd1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1263fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557357fc8e50 con 0x557357fc5ea0 2024-03-16T12:21:25.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1263fe2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557357ee0e20 con 0x557357fc0d30 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1263fe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557357fc8fd0 con 0x557357fc0340 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1263fe2740 10 client.0 _ll_register_callbacks cb 0x557357ee07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51320/0 (socket says 172.21.15.98:51320) 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f125a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 -- 172.21.15.98:0/677393795 learned_addr learned my addr 172.21.15.98:0/677393795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f125a800640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc5ea0 0x557357fd1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 -- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 msgr2=0x557357fca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 -- 172.21.15.98:0/677393795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc5ea0 msgr2=0x557357fd1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc5ea0 0x557357fd1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 -- 172.21.15.98:0/677393795 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557357c392d0 con 0x557357fc0d30 2024-03-16T12:21:25.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f125a800640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f1259400640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1248009900 tx=0x7f1248004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f2c7086808a53d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.199+0000 7f123be00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248005d50 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1248005230 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f125400f9a0 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1248005430 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 10 client.25364 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557357d036b0 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125400f9a0 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f12480039b0 con 0x557357fc0d30 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 10 client.25364 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 10 client.25364 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 20 client.25364 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:25.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557357fd7290 0x557357fd8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 4 client.25364 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.200+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557357f822b0 con 0x557357fd7290 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.201+0000 7f125a800640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557357fd7290 0x557357fd8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.201+0000 7f125a800640 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557357fd7290 0x557357fd8700 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd4e1aa0d037789f server_cookie=ef42d32f5b6621ca in_seq=0 out_seq=0 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.201+0000 7f123be00640 10 client.25364 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f123be00640 1 -- 172.21.15.98:0/677393795 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557357f822b0 con 0x557357fd7290 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f123be00640 10 client.25364 handle_command_reply: tid=0 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 client.25364 shutdown 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 msgr2=0x557357fca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1248009900 tx=0x7f1248004270 comp rx=0 tx=0).stop 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 shutdown_connections 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557357fd7290 0x557357fd8700 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc5ea0 0x557357fd1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fca9d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 --2- 172.21.15.98:0/677393795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc0340 0x557357fca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 >> 172.21.15.98:0/677393795 conn(0x557357fbb8b0 msgr2=0x557357fbaa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f123be00640 0 client.25364 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:21:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 shutdown_connections 2024-03-16T12:21:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 1 -- 172.21.15.98:0/677393795 wait complete. 2024-03-16T12:21:25.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.202+0000 7f1263fe2740 20 client.25364 trim_cache size 0 max 16384 2024-03-16T12:21:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.221+0000 7f1263fe2740 1 Processor -- start 2024-03-16T12:21:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.221+0000 7f1263fe2740 1 -- start start 2024-03-16T12:21:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 0x557357fba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fba720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557357fd6a40 con 0x557357fc0d30 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557357fd7110 con 0x557357fc0340 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1263fe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557357fd77e0 con 0x557357fc5ea0 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35772/0 (socket says 172.21.15.98:35772) 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 -- 172.21.15.98:0/360429516 learned_addr learned my addr 172.21.15.98:0/360429516 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f125a800640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 0x557357fba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f125a800640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fba720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 -- 172.21.15.98:0/360429516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 msgr2=0x557357fba720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fba720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 -- 172.21.15.98:0/360429516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 msgr2=0x557357fba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 0x557357fba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f1259400640 1 -- 172.21.15.98:0/360429516 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557357fd4cc0 con 0x557357fc0d30 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.222+0000 7f125a800640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 0x557357fba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1259400640 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1248001eb0 tx=0x7f1248005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c259f5a636bbe2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1246a00640 1 -- 172.21.15.98:0/360429516 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248005d50 con 0x557357fc0d30 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1246a00640 1 -- 172.21.15.98:0/360429516 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1248046e50 con 0x557357fc0d30 2024-03-16T12:21:25.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1246a00640 1 -- 172.21.15.98:0/360429516 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248003b30 con 0x557357fc0d30 2024-03-16T12:21:25.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 -- 172.21.15.98:0/360429516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 msgr2=0x557357fd19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1248001eb0 tx=0x7f1248005cb0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 -- 172.21.15.98:0/360429516 shutdown_connections 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fba720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0d30 0x557357fd19d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 --2- 172.21.15.98:0/360429516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0340 0x557357fba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.223+0000 7f1263fe2740 1 -- 172.21.15.98:0/360429516 >> 172.21.15.98:0/360429516 conn(0x557357fbb8b0 msgr2=0x557357fb6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- 172.21.15.98:0/360429516 shutdown_connections 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- 172.21.15.98:0/360429516 wait complete. 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- start start 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 Processor -- start 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 0x557357fcd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fd0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557357fd8580 con 0x557357fc0340 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557357fd7eb0 con 0x557357fc0d30 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557357fd8c50 con 0x557357fc5ea0 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.224+0000 7f1263fe2740 10 client.0 _ll_register_callbacks cb 0x7f124802c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f125a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 0x557357fcd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51346/0 (socket says 172.21.15.98:51346) 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 -- 172.21.15.98:0/2120222146 learned_addr learned my addr 172.21.15.98:0/2120222146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f125a800640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fd0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 -- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 msgr2=0x557357fd0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fd0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 -- 172.21.15.98:0/2120222146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 msgr2=0x557357fcd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 0x557357fcd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 -- 172.21.15.98:0/2120222146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557357fd0e70 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f125a800640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 0x557357fcd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1259400640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1248009340 tx=0x7f12480035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd63b9cbc6eaef05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f123be00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248005d50 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1248046e50 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.225+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1248003a30 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1248009520 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 10 client.25370 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55735803d210 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1248046e50 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1248034440 con 0x557357fc0d30 2024-03-16T12:21:25.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 10 client.25370 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 10 client.25370 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 20 client.25370 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557358041b10 0x557358040820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 4 client.25370 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "status"}) -- 0x557357f32190 con 0x557358041b10 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.226+0000 7f125a800640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557358041b10 0x557358040820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f125a800640 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557358041b10 0x557358040820 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5dd6ea9e1a6c979 server_cookie=853916f339414f70 in_seq=0 out_seq=0 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f123be00640 10 client.25370 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f123be00640 1 -- 172.21.15.98:0/2120222146 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557357f32190 con 0x557358041b10 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f123be00640 10 client.25370 handle_command_reply: tid=0 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 client.25370 shutdown 2024-03-16T12:21:25.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 msgr2=0x557357fcd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1248009340 tx=0x7f12480035f0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 shutdown_connections 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x557358041b10 0x557358040820 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557357fc5ea0 0x557357fd0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f123be00640 0 client.25370 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557357fc0d30 0x557357fcd860 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 --2- 172.21.15.98:0/2120222146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557357fc0340 0x557357fcd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 >> 172.21.15.98:0/2120222146 conn(0x557357fbb8b0 msgr2=0x557357fba1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.227+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 shutdown_connections 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f1263fe2740 1 -- 172.21.15.98:0/2120222146 wait complete. 2024-03-16T12:21:25.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f1263fe2740 20 client.25370 trim_cache size 0 max 16384 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24553, 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 139.331196543, 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:21:25.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:21:25.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:21:25.217 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 182.03007585099999 2024-03-16T12:21:25.217 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:21:25.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 msgr2=0x7f12400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 0x7f12400a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f124c0cc8c0 tx=0x7f124c0d2bc0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 msgr2=0x7f125c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f124c00e9c0 tx=0x7f124c00edc0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 shutdown_connections 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f12400a2800 0x7f12400a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c105490 0x7f125c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c104ad0 0x7f125c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 --2- 172.21.15.98:0/3830786328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125c1038e0 0x7f125c199bb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 >> 172.21.15.98:0/3830786328 conn(0x7f125c0fee50 msgr2=0x7f125c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 shutdown_connections 2024-03-16T12:21:25.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.228+0000 7f125b200640 1 -- 172.21.15.98:0/3830786328 wait complete. 2024-03-16T12:21:25.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T12:21:25.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:25 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3268912810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:25 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/383913354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:21:25.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:21:25.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:25.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:25 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3268912810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:25 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/383913354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:21:25.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:21:25.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:25 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3268912810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:25 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/383913354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:25.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:21:25.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:21:25.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:21:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 -- 172.21.15.98:0/3737526426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 msgr2=0x7f51280746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 --2- 172.21.15.98:0/3737526426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f51280746c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5114009890 tx=0x7f511402bb70 comp rx=0 tx=0).stop 2024-03-16T12:21:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 -- 172.21.15.98:0/3737526426 shutdown_connections 2024-03-16T12:21:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 --2- 172.21.15.98:0/3737526426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 0x7f51281101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 --2- 172.21.15.98:0/3737526426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f51280746c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 --2- 172.21.15.98:0/3737526426 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 0x7f5128073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.450+0000 7f512cc00640 1 -- 172.21.15.98:0/3737526426 >> 172.21.15.98:0/3737526426 conn(0x7f51280fcd60 msgr2=0x7f51280ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.451+0000 7f512cc00640 1 -- 172.21.15.98:0/3737526426 shutdown_connections 2024-03-16T12:21:25.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.451+0000 7f512cc00640 1 -- 172.21.15.98:0/3737526426 wait complete. 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.451+0000 7f512cc00640 1 Processor -- start 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 -- start start 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 0x7f512819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 0x7f51281a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5128113a60 con 0x7f5128075820 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51281138e0 con 0x7f5128074240 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f512cc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5128113be0 con 0x7f512810ddc0 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 0x7f512819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51358/0 (socket says 172.21.15.98:51358) 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 -- 172.21.15.98:0/2187260356 learned_addr learned my addr 172.21.15.98:0/2187260356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5126800640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 0x7f51281a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 -- 172.21.15.98:0/2187260356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 msgr2=0x7f51281a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 0x7f51281a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 -- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 msgr2=0x7f512819e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.452+0000 7f5125e00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 0x7f512819e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f5125e00640 1 -- 172.21.15.98:0/2187260356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51281a5d80 con 0x7f5128074240 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f5125e00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f511800d950 tx=0x7f511800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f511ea00640 1 -- 172.21.15.98:0/2187260356 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118017070 con 0x7f5128074240 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f511ea00640 1 -- 172.21.15.98:0/2187260356 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5118009c30 con 0x7f5128074240 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f511ea00640 1 -- 172.21.15.98:0/2187260356 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511800f6b0 con 0x7f5128074240 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51281a6070 con 0x7f5128074240 2024-03-16T12:21:25.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.453+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51281a6540 con 0x7f5128074240 2024-03-16T12:21:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.456+0000 7f512f9e2740 1 Processor -- start 2024-03-16T12:21:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.456+0000 7f511ea00640 1 -- 172.21.15.98:0/2187260356 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51180109d0 con 0x7f5128074240 2024-03-16T12:21:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.456+0000 7f512f9e2740 1 -- start start 2024-03-16T12:21:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 0x5625178f7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178fccc0 0x5625178ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56251785bc50 con 0x5625178f7160 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5625178049d0 con 0x5625178f7b50 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f512f9e2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562517857cf0 con 0x5625178fccc0 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 0x5625178f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5125e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5125400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178fccc0 0x5625178ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5125e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51360/0 (socket says 172.21.15.98:51360) 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5125e00640 1 -- 172.21.15.98:0/1616980344 learned_addr learned my addr 172.21.15.98:0/1616980344 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 -- 172.21.15.98:0/1616980344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178fccc0 msgr2=0x5625178ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178fccc0 0x5625178ff150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 -- 172.21.15.98:0/1616980344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 msgr2=0x5625178fc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5126800640 1 -- 172.21.15.98:0/1616980344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56251781ef10 con 0x5625178f7160 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.457+0000 7f5125e00640 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f5126800640 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 0x5625178f7580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f512000e450 tx=0x7f51200095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cda1ccc73c957dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f5113000640 1 -- 172.21.15.98:0/1616980344 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120004b20 con 0x5625178f7160 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f5113000640 1 -- 172.21.15.98:0/1616980344 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5120004cc0 con 0x5625178f7160 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f511ea00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 0x7f510c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f5113000640 1 -- 172.21.15.98:0/1616980344 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200056c0 con 0x5625178f7160 2024-03-16T12:21:25.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f511ea00640 1 -- 172.21.15.98:0/2187260356 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51180ce900 con 0x7f5128074240 2024-03-16T12:21:25.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.458+0000 7f5125e00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 0x7f510c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- 172.21.15.98:0/1616980344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 msgr2=0x5625178f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 0x5625178f7580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f512000e450 tx=0x7f51200095c0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- 172.21.15.98:0/1616980344 shutdown_connections 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178fccc0 0x5625178ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625178fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 --2- 172.21.15.98:0/1616980344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7160 0x5625178f7580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- 172.21.15.98:0/1616980344 >> 172.21.15.98:0/1616980344 conn(0x5625178f26d0 msgr2=0x5625178f2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- 172.21.15.98:0/1616980344 shutdown_connections 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- 172.21.15.98:0/1616980344 wait complete. 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f5125e00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 0x7f510c0a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f51180cc0a0 tx=0x7f51180d4550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.459+0000 7f512f9e2740 1 -- start start 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 Processor -- start 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178fccc0 0x5625179087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5625178ffc70 con 0x5625178f7b50 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562517817e20 con 0x5625178fccc0 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625178ffdf0 con 0x5625178f7160 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f512f9e2740 10 client.0 _ll_register_callbacks cb 0x5625178177f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5125400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5126800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5126800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48888/0 (socket says 172.21.15.98:48888) 2024-03-16T12:21:25.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5126800640 1 -- 172.21.15.98:0/2743461321 learned_addr learned my addr 172.21.15.98:0/2743461321 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5125400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35800/0 (socket says 172.21.15.98:35800) 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.460+0000 7f5126800640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178fccc0 0x5625179087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 -- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178fccc0 msgr2=0x5625179087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178fccc0 0x5625179087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 -- 172.21.15.98:0/2743461321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 msgr2=0x5625179017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 -- 172.21.15.98:0/2743461321 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625175702d0 con 0x5625178f7160 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5125400640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5126800640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5120005fd0 tx=0x7f5120009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc0c49a9e553a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5107400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120005670 con 0x5625178f7160 2024-03-16T12:21:25.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5120003970 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5120004cc0 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5120033050 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f512f9e2740 10 client.25372 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.461+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56251763a6b0 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5120004cc0 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f512000b040 con 0x5625178f7160 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 10 client.25372 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 10 client.25372 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 20 client.25372 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56251790dbe0 0x56251790f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 4 client.25372 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:21:25.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625178b92b0 con 0x56251790dbe0 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.462+0000 7f5125400640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56251790dbe0 0x56251790f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.463+0000 7f5125400640 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56251790dbe0 0x56251790f050 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3872db979792bd7e server_cookie=11501aed7eefa490 in_seq=0 out_seq=0 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.463+0000 7f5107400640 10 client.25372 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.463+0000 7f5107400640 1 -- 172.21.15.98:0/2743461321 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625178b92b0 con 0x56251790dbe0 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f5107400640 10 client.25372 handle_command_reply: tid=0 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 client.25372 shutdown 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 msgr2=0x562517901270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5120005fd0 tx=0x7f5120009780 comp rx=0 tx=0).stop 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 shutdown_connections 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178fccc0 0x5625179087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178f7b50 0x5625179017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56251790dbe0 0x56251790f050 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2743461321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517901270 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 >> 172.21.15.98:0/2743461321 conn(0x5625178f26d0 msgr2=0x5625178f1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f5107400640 0 client.25372 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 shutdown_connections 2024-03-16T12:21:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 1 -- 172.21.15.98:0/2743461321 wait complete. 2024-03-16T12:21:25.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.464+0000 7f512f9e2740 20 client.25372 trim_cache size 0 max 16384 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.483+0000 7f512f9e2740 1 Processor -- start 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.483+0000 7f512f9e2740 1 -- start start 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625179087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x5625179013c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56251790d390 con 0x5625178fccc0 2024-03-16T12:21:25.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56251790da60 con 0x5625178f7b50 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f512f9e2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56251790e130 con 0x5625178f7160 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5125400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625179087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48890/0 (socket says 172.21.15.98:48890) 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 -- 172.21.15.98:0/2873351764 learned_addr learned my addr 172.21.15.98:0/2873351764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x5625179013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 -- 172.21.15.98:0/2873351764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 msgr2=0x5625179087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625179087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 -- 172.21.15.98:0/2873351764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 msgr2=0x5625179013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x5625179013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.484+0000 7f5126800640 1 -- 172.21.15.98:0/2873351764 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625178c8750 con 0x5625178f7160 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.485+0000 7f5126800640 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5120004250 tx=0x7f5120000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65eaed36ad4d260a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.485+0000 7f5112600640 1 -- 172.21.15.98:0/2873351764 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120029070 con 0x5625178f7160 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.485+0000 7f5112600640 1 -- 172.21.15.98:0/2873351764 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5120004cc0 con 0x5625178f7160 2024-03-16T12:21:25.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.485+0000 7f5112600640 1 -- 172.21.15.98:0/2873351764 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200045d0 con 0x5625178f7160 2024-03-16T12:21:25.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- 172.21.15.98:0/2873351764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 msgr2=0x562517900e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5120004250 tx=0x7f5120000f40 comp rx=0 tx=0).stop 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- 172.21.15.98:0/2873351764 shutdown_connections 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x5625179013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x5625179087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 --2- 172.21.15.98:0/2873351764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x562517900e80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- 172.21.15.98:0/2873351764 >> 172.21.15.98:0/2873351764 conn(0x5625178f26d0 msgr2=0x5625178f2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- 172.21.15.98:0/2873351764 shutdown_connections 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- 172.21.15.98:0/2873351764 wait complete. 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 -- start start 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.486+0000 7f512f9e2740 1 Processor -- start 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x5625179067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x562517971e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56251790eed0 con 0x5625178fccc0 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56251790e800 con 0x5625178f7b50 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56251790f5a0 con 0x5625178f7160 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f512f9e2740 10 client.0 _ll_register_callbacks cb 0x7f5120004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5126800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x5625179067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5126800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x562517971e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:51392/0 (socket says 172.21.15.98:51392) 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 -- 172.21.15.98:0/4044259628 learned_addr learned my addr 172.21.15.98:0/4044259628 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 -- 172.21.15.98:0/4044259628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 msgr2=0x5625179067a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x5625179067a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 -- 172.21.15.98:0/4044259628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 msgr2=0x562517971e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x562517971e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.487+0000 7f5125400640 1 -- 172.21.15.98:0/4044259628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625179723c0 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5125400640 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5114009250 tx=0x7f51140323f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af7a62d359f58cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5107400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511404c070 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5114004040 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5120004cc0 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5114058050 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f512f9e2740 10 client.25388 fetch_fsmap learned FSMap version 23 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5625179736d0 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.488+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5120004cc0 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f5114032640 con 0x5625178f7b50 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 10 client.25388 fetch_fsmap finished waiting for FSMap version 23 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 10 client.25388 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 20 client.25388 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x562517978280 0x562517976f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 4 client.25388 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "status"}) -- 0x562517869190 con 0x562517978280 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f5126800640 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x562517978280 0x562517976f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f5126800640 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x562517978280 0x562517976f90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266005ecbaca6686 server_cookie=4c35a7d57c6618fc in_seq=0 out_seq=0 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.489+0000 7f5107400640 10 client.25388 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f5107400640 1 -- 172.21.15.98:0/4044259628 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562517869190 con 0x562517978280 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f5107400640 10 client.25388 handle_command_reply: tid=0 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 client.25388 shutdown 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 msgr2=0x56251796f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5114009250 tx=0x7f51140323f0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 shutdown_connections 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5625178fccc0 0x562517971e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5625178f7b50 0x56251796f930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x562517978280 0x562517976f90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 --2- 172.21.15.98:0/4044259628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625178f7160 0x5625179067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 >> 172.21.15.98:0/4044259628 conn(0x5625178f26d0 msgr2=0x562517900e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f5107400640 0 client.25388 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 shutdown_connections 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 1 -- 172.21.15.98:0/4044259628 wait complete. 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.490+0000 7f512f9e2740 20 client.25388 trim_cache size 0 max 16384 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "e9cfddf2-e38d-11ee-95c9-87774f69a715", 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24599, 2024-03-16T12:21:25.476 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 164.46533326599999, 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 23, 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 171.138351586 2024-03-16T12:21:25.477 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 msgr2=0x7f510c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 0x7f510c0a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f51180cc0a0 tx=0x7f51180d4550 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 msgr2=0x7f512819e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f511800d950 tx=0x7f511800de10 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 shutdown_connections 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f510c0a2880 0x7f510c0a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f512810ddc0 0x7f51281a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5128075820 0x7f512819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 --2- 172.21.15.98:0/2187260356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5128074240 0x7f512819e160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 >> 172.21.15.98:0/2187260356 conn(0x7f51280fcd60 msgr2=0x7f5128101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 shutdown_connections 2024-03-16T12:21:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:25.491+0000 7f512cc00640 1 -- 172.21.15.98:0/2187260356 wait complete. 2024-03-16T12:21:25.490 INFO:teuthology.run_tasks:Running task kclient... 2024-03-16T12:21:25.504 INFO:tasks.kclient:Mounting kernel clients... 2024-03-16T12:21:25.504 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-16T12:21:25.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:25.504 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-16T12:21:25.557 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-16T12:21:25.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:25.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T12:21:25.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:25.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-16T12:21:25.788 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-16T12:21:25.789 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T12:21:25.789 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:25.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T12:21:26.222 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:21:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:26 smithi153 ceph-mon[36901]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:26.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:26 smithi173 ceph-mon[36632]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:26.373 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:26 smithi098 ceph-mon[31633]: pgmap v402: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:21:28.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:28 smithi098 ceph-mon[31633]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:28.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:28 smithi153 ceph-mon[36901]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:28.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:28 smithi173 ceph-mon[36632]: pgmap v403: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 -- 172.21.15.98:0/416534317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 msgr2=0x7f7c740748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:28.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 --2- 172.21.15.98:0/416534317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c740748e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009990 tx=0x7f7c64009ca0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 -- 172.21.15.98:0/416534317 shutdown_connections 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 --2- 172.21.15.98:0/416534317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c740748e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 --2- 172.21.15.98:0/416534317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 0x7f7c74115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 --2- 172.21.15.98:0/416534317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 0x7f7c74112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 -- 172.21.15.98:0/416534317 >> 172.21.15.98:0/416534317 conn(0x7f7c74079430 msgr2=0x7f7c74079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 -- 172.21.15.98:0/416534317 shutdown_connections 2024-03-16T12:21:28.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.654+0000 7f7c79e00700 1 -- 172.21.15.98:0/416534317 wait complete. 2024-03-16T12:21:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 Processor -- start 2024-03-16T12:21:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 -- start start 2024-03-16T12:21:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 0x7f7c7419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 0x7f7c741a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.656+0000 7f7c79e00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74118ce0 con 0x7f7c741134a0 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c79e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74118b60 con 0x7f7c74074500 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c79e00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74118e60 con 0x7f7c74074e20 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50576/0 (socket says 172.21.15.98:50576) 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 -- 172.21.15.98:0/3490114 learned_addr learned my addr 172.21.15.98:0/3490114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72000700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 0x7f7c7419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c73400700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 0x7f7c741a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 -- 172.21.15.98:0/3490114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 msgr2=0x7f7c7419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 0x7f7c7419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 -- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 msgr2=0x7f7c741a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 0x7f7c741a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 -- 172.21.15.98:0/3490114 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c741a4160 con 0x7f7c74074500 2024-03-16T12:21:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.657+0000 7f7c72a00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.658+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64027070 con 0x7f7c74074500 2024-03-16T12:21:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.658+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c640056b0 con 0x7f7c74074500 2024-03-16T12:21:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.658+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64017650 con 0x7f7c74074500 2024-03-16T12:21:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.658+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741a43f0 con 0x7f7c74074500 2024-03-16T12:21:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.658+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741abcd0 con 0x7f7c74074500 2024-03-16T12:21:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.659+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c74068f90 con 0x7f7c74074500 2024-03-16T12:21:28.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.661+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c6402a070 con 0x7f7c74074500 2024-03-16T12:21:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.662+0000 7f7c78a00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 0x7f7c580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.662+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c640cfd60 con 0x7f7c74074500 2024-03-16T12:21:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.662+0000 7f7c72000700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 0x7f7c580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.663+0000 7f7c72000700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 0x7f7c580a4dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7c7410fd20 tx=0x7f7c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:28.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.671+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c6405e740 con 0x7f7c74074500 2024-03-16T12:21:28.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.943+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7c741abea0 con 0x7f7c74074500 2024-03-16T12:21:28.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.943+0000 7f7c78a00700 1 -- 172.21.15.98:0/3490114 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+88 (secure 0 0 0) 0x7f7c64013070 con 0x7f7c74074500 2024-03-16T12:21:28.925 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 msgr2=0x7f7c580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 0x7f7c580a4dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7c7410fd20 tx=0x7f7c60009260 comp rx=0 tx=0).stop 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 msgr2=0x7f7c7419f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64005590 comp rx=0 tx=0).stop 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 shutdown_connections 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7c580a2910 0x7f7c580a4dd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c74074500 0x7f7c7419f150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c741134a0 0x7f7c741a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 --2- 172.21.15.98:0/3490114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c74074e20 0x7f7c7419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 >> 172.21.15.98:0/3490114 conn(0x7f7c74079430 msgr2=0x7f7c74112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.946+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 shutdown_connections 2024-03-16T12:21:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:28.947+0000 7f7c79e00700 1 -- 172.21.15.98:0/3490114 wait complete. 2024-03-16T12:21:29.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:29 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3490114' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:21:29.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:29 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3490114' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:21:29.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:29 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3490114' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T12:21:29.875 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T12:21:29.876 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T12:21:29.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:29.876 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-03-16T12:21:29.930 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-03-16T12:21:29.931 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41938sec preferred_lft 41938sec 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-03-16T12:21:29.932 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-03-16T12:21:29.933 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T12:21:29.933 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:21:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:30.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:30 smithi098 ceph-mon[31633]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:30.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:30.225 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T12:21:30.303 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-16T12:21:30.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:30.308 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-03-16T12:21:30.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:30 smithi173 ceph-mon[36632]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:30.360 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-03-16T12:21:30.360 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T12:21:30.360 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T12:21:30.361 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T12:21:30.361 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T12:21:30.362 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:21:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:30.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:30 smithi153 ceph-mon[36901]: pgmap v404: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:21:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:30.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:30.611 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-16T12:21:30.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:30.663 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-03-16T12:21:30.717 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:21:30.717 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:21:30.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T12:21:30.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T12:21:30.718 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:21:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:30.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:30.917 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T12:21:30.918 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:21:31.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:31.108 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:31 smithi098 ceph-mon[31633]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:21:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:31.238 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T12:21:31.238 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-03-16T12:21:31.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-03-16T12:21:31.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T12:21:31.239 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-03-16T12:21:31.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:31.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:31.464 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T12:21:31.464 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:31.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:31 smithi153 ceph-mon[36901]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:21:31.506 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T12:21:31.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T12:21:31.507 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:31.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:31 smithi173 ceph-mon[36632]: pgmap v405: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:21:31.604 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-16T12:21:31.604 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-16T12:21:31.604 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/dedb0f08-2fbc-4337-afa2-6e387c4edd3d 2024-03-16T12:21:31.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:31.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/dedb0f08-2fbc-4337-afa2-6e387c4edd3d /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-16T12:21:31.744 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-16T12:21:31.745 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-16T12:21:31.745 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-03-16T12:21:32.411 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-16T12:21:32.411 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-16T12:21:32.411 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-03-16T12:21:32.411 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.153:3300,172.21.15.173:3300" 2024-03-16T12:21:32.411 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:3300,172.21.15.153:3300,172.21.15.173:3300:/volumes/qa/sv_1/dedb0f08-2fbc-4337-afa2-6e387c4edd3d 2024-03-16T12:21:32.412 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=e9cfddf2-e38d-11ee-95c9-87774f69a715" will pass to kernel 2024-03-16T12:21:32.417 INFO:teuthology.orchestra.run.smithi098.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-16T12:21:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: You just mounted a file system that supports labels which does not 2024-03-16T12:21:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-16T12:21:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: applications will generate AVC messages and not be allowed access to 2024-03-16T12:21:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-16T12:21:32.423 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-16T12:21:32.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:32.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:32.512 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T12:21:32.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T12:21:32.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:32.633 INFO:teuthology.orchestra.run.smithi098.stdout:client25397 2024-03-16T12:21:32.634 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:21:32.634 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:21:32.684 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T12:21:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T12:21:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:32.853 INFO:teuthology.orchestra.run.smithi098.stdout:client25397 2024-03-16T12:21:32.853 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T12:21:32.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T12:21:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:32.969 INFO:teuthology.orchestra.run.smithi098.stdout:client25397 2024-03-16T12:21:32.970 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715.client25397/status') 2024-03-16T12:21:33.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T12:21:33.116 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.25397 (3)172.21.15.98:0/2476611614 2024-03-16T12:21:33.117 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-03-16T12:21:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-03-16T12:21:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-03-16T12:21:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:71 bytes copied, 6.793e-05 s, 1.0 MB/s 2024-03-16T12:21:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T12:21:33.130 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T12:21:33.141 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T12:21:33.141 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:21:33.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-16T12:21:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 -- 172.21.15.98:0/2878462627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 msgr2=0x7fa6d0101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 --2- 172.21.15.98:0/2878462627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0101a70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009900 tx=0x7fa6c002bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 -- 172.21.15.98:0/2878462627 shutdown_connections 2024-03-16T12:21:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 --2- 172.21.15.98:0/2878462627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 0x7fa6d0109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 --2- 172.21.15.98:0/2878462627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 --2- 172.21.15.98:0/2878462627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0101a70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 -- 172.21.15.98:0/2878462627 >> 172.21.15.98:0/2878462627 conn(0x7fa6d00fcbc0 msgr2=0x7fa6d00ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 -- 172.21.15.98:0/2878462627 shutdown_connections 2024-03-16T12:21:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.364+0000 7fa6cf200640 1 -- 172.21.15.98:0/2878462627 wait complete. 2024-03-16T12:21:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.366+0000 7fa6cf200640 1 Processor -- start 2024-03-16T12:21:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.366+0000 7fa6cf200640 1 -- start start 2024-03-16T12:21:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.366+0000 7fa6cf200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.366+0000 7fa6cf200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cf200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 0x7fa6d0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cf200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d010d2d0 con 0x7fa6d0103200 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cf200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d010d150 con 0x7fa6d0101650 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cf200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d010d450 con 0x7fa6d0102840 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50600/0 (socket says 172.21.15.98:50600) 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 -- 172.21.15.98:0/3031086193 learned_addr learned my addr 172.21.15.98:0/3031086193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6c7e00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6ce800640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 0x7fa6d0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 -- 172.21.15.98:0/3031086193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 msgr2=0x7fa6d0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 -- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 msgr2=0x7fa6d0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 0x7fa6d0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6cde00640 1 -- 172.21.15.98:0/3031086193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d00749a0 con 0x7fa6d0101650 2024-03-16T12:21:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.367+0000 7fa6c7e00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6cde00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0041070 con 0x7fa6d0101650 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c0004450 con 0x7fa6d0101650 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0034440 con 0x7fa6d0101650 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d01a1a60 con 0x7fa6d0101650 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.368+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d01a1fb0 con 0x7fa6d0101650 2024-03-16T12:21:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.369+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d01089f0 con 0x7fa6d0101650 2024-03-16T12:21:33.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.371+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa6c0044070 con 0x7fa6d0101650 2024-03-16T12:21:33.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.372+0000 7fa6c6a00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 0x7fa6b00a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:33.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.372+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa6c00f0880 con 0x7fa6d0101650 2024-03-16T12:21:33.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.372+0000 7fa6c7e00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 0x7fa6b00a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:33.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.373+0000 7fa6c7e00640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 0x7fa6b00a4cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0075300 tx=0x7fa6b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.376+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6c00bd000 con 0x7fa6d0101650 2024-03-16T12:21:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.618+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fa6d00623e0 con 0x7fa6d0101650 2024-03-16T12:21:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.909+0000 7fa6c6a00640 1 -- 172.21.15.98:0/3031086193 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7fa6c00c10a0 con 0x7fa6d0101650 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.912+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 msgr2=0x7fa6b00a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.912+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 0x7fa6b00a4cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0075300 tx=0x7fa6b8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.912+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 msgr2=0x7fa6d0075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.912+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c002c640 comp rx=0 tx=0).stop 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 shutdown_connections 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa6b00a27f0 0x7fa6b00a4cb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d0103200 0x7fa6d0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d0102840 0x7fa6d0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 --2- 172.21.15.98:0/3031086193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d0101650 0x7fa6d0075590 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 >> 172.21.15.98:0/3031086193 conn(0x7fa6d00fcbc0 msgr2=0x7fa6d0105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 shutdown_connections 2024-03-16T12:21:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:33.913+0000 7fa6cf200640 1 -- 172.21.15.98:0/3031086193 wait complete. 2024-03-16T12:21:33.916 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T12:21:33.927 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-16T12:21:33.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 -- 172.21.15.98:0/2513998294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec1055b0 msgr2=0x7f26ec10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 --2- 172.21.15.98:0/2513998294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec1055b0 0x7f26ec10be40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009ca0 tx=0x7f26e402c100 comp rx=0 tx=0).stop 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 -- 172.21.15.98:0/2513998294 shutdown_connections 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 --2- 172.21.15.98:0/2513998294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec1055b0 0x7f26ec10be40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 --2- 172.21.15.98:0/2513998294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec104bf0 0x7f26ec105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 --2- 172.21.15.98:0/2513998294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 0x7f26ec103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 -- 172.21.15.98:0/2513998294 >> 172.21.15.98:0/2513998294 conn(0x7f26ec0fef90 msgr2=0x7f26ec1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:34.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 -- 172.21.15.98:0/2513998294 shutdown_connections 2024-03-16T12:21:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.208+0000 7f26f2000640 1 -- 172.21.15.98:0/2513998294 wait complete. 2024-03-16T12:21:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.209+0000 7f26f2000640 1 Processor -- start 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.209+0000 7f26f2000640 1 -- start start 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 0x7f26ec199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 0x7f26ec1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10f680 con 0x7f26ec1055b0 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10f500 con 0x7f26ec103a00 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26f2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10f800 con 0x7f26ec104bf0 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 0x7f26ec1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44938/0 (socket says 172.21.15.98:44938) 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 0x7f26ec199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 -- 172.21.15.98:0/766993891 learned_addr learned my addr 172.21.15.98:0/766993891 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 -- 172.21.15.98:0/766993891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 msgr2=0x7f26ec199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 0x7f26ec199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 -- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 msgr2=0x7f26ec1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 0x7f26ec1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26eaa00640 1 -- 172.21.15.98:0/766993891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec1a1870 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.210+0000 7f26ebe00640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 0x7f26ec1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26eaa00640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00e9f0 tx=0x7f26dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00cd80 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc010910 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec1a1b60 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc018990 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.211+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec1a20b0 con 0x7f26ec104bf0 2024-03-16T12:21:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.212+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec10ad20 con 0x7f26ec104bf0 2024-03-16T12:21:34.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.214+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f26dc010ac0 con 0x7f26ec104bf0 2024-03-16T12:21:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.215+0000 7f26e3e00640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 0x7f26d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.215+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f26dc0cf200 con 0x7f26ec104bf0 2024-03-16T12:21:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.215+0000 7f26eb400640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 0x7f26d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.216+0000 7f26eb400640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 0x7f26d00a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26ec19b130 tx=0x7f26d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.219+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26dc09b890 con 0x7f26ec104bf0 2024-03-16T12:21:34.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:33 smithi098 ceph-mon[31633]: pgmap v406: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:34.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:33 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3031086193' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:33 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:33 smithi153 ceph-mon[36901]: pgmap v406: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:34.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:33 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3031086193' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:33 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:33 smithi173 ceph-mon[36632]: pgmap v406: 129 pgs: 129 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:21:34.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:33 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3031086193' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:33 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T12:21:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.420+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26ec0623e0 con 0x7f26ec104bf0 2024-03-16T12:21:34.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.422+0000 7f26e3e00640 1 -- 172.21.15.98:0/766993891 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f26dc09f930 con 0x7f26ec104bf0 2024-03-16T12:21:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:34.404 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:34.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.424+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 msgr2=0x7f26d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.424+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 0x7f26d00a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26ec19b130 tx=0x7f26d8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:34.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.424+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 msgr2=0x7f26ec19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.424+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00e9f0 tx=0x7f26dc00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 shutdown_connections 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f26d00a2800 0x7f26d00a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26ec1055b0 0x7f26ec1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26ec104bf0 0x7f26ec19a1a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 --2- 172.21.15.98:0/766993891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26ec103a00 0x7f26ec199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 >> 172.21.15.98:0/766993891 conn(0x7f26ec0fef90 msgr2=0x7f26ec107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 shutdown_connections 2024-03-16T12:21:34.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.425+0000 7f26f2000640 1 -- 172.21.15.98:0/766993891 wait complete. 2024-03-16T12:21:34.427 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:21:34.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:21:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.664+0000 7f77a3e00640 1 -- 172.21.15.98:0/3443219196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c1074e0 msgr2=0x7f779c109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.664+0000 7f77a3e00640 1 --2- 172.21.15.98:0/3443219196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c1074e0 0x7f779c109930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7794009900 tx=0x7f779402bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:34.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 -- 172.21.15.98:0/3443219196 shutdown_connections 2024-03-16T12:21:34.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 --2- 172.21.15.98:0/3443219196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c1074e0 0x7f779c109930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 --2- 172.21.15.98:0/3443219196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c104bb0 0x7f779c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 --2- 172.21.15.98:0/3443219196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c102280 0x7f779c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 -- 172.21.15.98:0/3443219196 >> 172.21.15.98:0/3443219196 conn(0x7f779c0fbc90 msgr2=0x7f779c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:34.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 -- 172.21.15.98:0/3443219196 shutdown_connections 2024-03-16T12:21:34.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.665+0000 7f77a3e00640 1 -- 172.21.15.98:0/3443219196 wait complete. 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 Processor -- start 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 -- start start 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c104bb0 0x7f779c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c1074e0 0x7f779c100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f779c10d480 con 0x7f779c1074e0 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f779c10d300 con 0x7f779c104bb0 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f779c10d600 con 0x7f779c102280 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.666+0000 7f77a1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44950/0 (socket says 172.21.15.98:44950) 2024-03-16T12:21:34.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 -- 172.21.15.98:0/4207526005 learned_addr learned my addr 172.21.15.98:0/4207526005 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a0a00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c104bb0 0x7f779c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c1074e0 0x7f779c100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 -- 172.21.15.98:0/4207526005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c104bb0 msgr2=0x7f779c100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c104bb0 0x7f779c100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 -- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c1074e0 msgr2=0x7f779c100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c1074e0 0x7f779c100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 -- 172.21.15.98:0/4207526005 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c101000 con 0x7f779c102280 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a1400640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f779000ebc0 tx=0x7f779000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790010630 con 0x7f779c102280 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779c1a1bb0 con 0x7f779c102280 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7790010cb0 con 0x7f779c102280 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.667+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790018bd0 con 0x7f779c102280 2024-03-16T12:21:34.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.668+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779c1a2150 con 0x7f779c102280 2024-03-16T12:21:34.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.669+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f779c068a20 con 0x7f779c102280 2024-03-16T12:21:34.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.671+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f77900107d0 con 0x7f779c102280 2024-03-16T12:21:34.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.672+0000 7f779a000640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 0x7f77840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:34.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.673+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77900cf360 con 0x7f779c102280 2024-03-16T12:21:34.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.673+0000 7f77a0a00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 0x7f77840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:34.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.673+0000 7f77a0a00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 0x7f77840a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77880095f0 tx=0x7f7788005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:34.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.674+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f779009b9f0 con 0x7f779c102280 2024-03-16T12:21:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.873+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f779c02d280 con 0x7f779c102280 2024-03-16T12:21:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.874+0000 7f779a000640 1 -- 172.21.15.98:0/4207526005 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f779c02d280 con 0x7f779c102280 2024-03-16T12:21:34.855 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:21:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 msgr2=0x7f77840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 0x7f77840a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77880095f0 tx=0x7f7788005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 msgr2=0x7f779c101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f779000ebc0 tx=0x7f779000c6a0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 shutdown_connections 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f77840a2880 0x7f77840a4d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f779c1074e0 0x7f779c100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f779c104bb0 0x7f779c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 --2- 172.21.15.98:0/4207526005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779c102280 0x7f779c101bf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 >> 172.21.15.98:0/4207526005 conn(0x7f779c0fbc90 msgr2=0x7f779c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 shutdown_connections 2024-03-16T12:21:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:34.877+0000 7f77a3e00640 1 -- 172.21.15.98:0/4207526005 wait complete. 2024-03-16T12:21:34.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:21:35.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 -- 172.21.15.98:0/2398503658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 msgr2=0x7ff38c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 --2- 172.21.15.98:0/2398503658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c102e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff374009900 tx=0x7ff37402bc50 comp rx=0 tx=0).stop 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 -- 172.21.15.98:0/2398503658 shutdown_connections 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 --2- 172.21.15.98:0/2398503658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 --2- 172.21.15.98:0/2398503658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c102e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 --2- 172.21.15.98:0/2398503658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 -- 172.21.15.98:0/2398503658 >> 172.21.15.98:0/2398503658 conn(0x7ff38c0fcd80 msgr2=0x7ff38c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 -- 172.21.15.98:0/2398503658 shutdown_connections 2024-03-16T12:21:35.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.106+0000 7ff392800640 1 -- 172.21.15.98:0/2398503658 wait complete. 2024-03-16T12:21:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.107+0000 7ff392800640 1 Processor -- start 2024-03-16T12:21:35.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 -- start start 2024-03-16T12:21:35.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c1a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10d450 con 0x7ff38c1029e0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10d2d0 con 0x7ff38c1033a0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff392800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c10d5d0 con 0x7ff38c1017f0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff382a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff38be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c1a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44978/0 (socket says 172.21.15.98:44978) 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.108+0000 7ff38b400640 1 -- 172.21.15.98:0/1567894680 learned_addr learned my addr 172.21.15.98:0/1567894680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 -- 172.21.15.98:0/1567894680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 msgr2=0x7ff38c1a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c1a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 -- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 msgr2=0x7ff38c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 -- 172.21.15.98:0/1567894680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c1a1a60 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38be00640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c1a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff38b400640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00e9c0 tx=0x7ff37c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00ccc0 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37c0035f0 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c1a1d50 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00ce60 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.109+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38c1a21c0 con 0x7ff38c1017f0 2024-03-16T12:21:35.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.111+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff38c1a2450 con 0x7ff38c1017f0 2024-03-16T12:21:35.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.112+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff37c0037a0 con 0x7ff38c1017f0 2024-03-16T12:21:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.113+0000 7ff388c00640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 0x7ff3700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.114+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff37c0cf180 con 0x7ff38c1017f0 2024-03-16T12:21:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.114+0000 7ff382a00640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 0x7ff3700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.114+0000 7ff382a00640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 0x7ff3700a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff37402c7d0 tx=0x7ff374005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.118+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff37c09be60 con 0x7ff38c1017f0 2024-03-16T12:21:35.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:34 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T12:21:35.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:34 smithi098 ceph-mon[31633]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:21:35.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:34 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/766993891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:35.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:34 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4207526005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:35.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:34 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T12:21:35.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:34 smithi153 ceph-mon[36901]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:21:35.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:34 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/766993891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:35.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:34 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4207526005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.317+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff38c1a2620 con 0x7ff38c1017f0 2024-03-16T12:21:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.317+0000 7ff388c00640 1 -- 172.21.15.98:0/1567894680 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7ff37c09b760 con 0x7ff38c1017f0 2024-03-16T12:21:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 msgr2=0x7ff3700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 0x7ff3700a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff37402c7d0 tx=0x7ff374005fd0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 msgr2=0x7ff38c199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00e9c0 tx=0x7ff37c00edc0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 shutdown_connections 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff3700a2800 0x7ff3700a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff38c1033a0 0x7ff38c1a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff38c1029e0 0x7ff38c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 --2- 172.21.15.98:0/1567894680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff38c1017f0 0x7ff38c199e40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 >> 172.21.15.98:0/1567894680 conn(0x7ff38c0fcd80 msgr2=0x7ff38c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.320+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 shutdown_connections 2024-03-16T12:21:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.321+0000 7ff392800640 1 -- 172.21.15.98:0/1567894680 wait complete. 2024-03-16T12:21:35.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:34 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T12:21:35.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:34 smithi173 ceph-mon[36632]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:21:35.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:34 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/766993891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:35.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:34 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4207526005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:35.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 -- 172.21.15.98:0/3403836370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19401099b0 msgr2=0x7f1940110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 --2- 172.21.15.98:0/3403836370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19401099b0 0x7f1940110240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1938009ca0 tx=0x7f193802c100 comp rx=0 tx=0).stop 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 -- 172.21.15.98:0/3403836370 shutdown_connections 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 --2- 172.21.15.98:0/3403836370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19401099b0 0x7f1940110240 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 --2- 172.21.15.98:0/3403836370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 0x7f1940109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 --2- 172.21.15.98:0/3403836370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940107e00 0x7f1940108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.546+0000 7f1945200640 1 -- 172.21.15.98:0/3403836370 >> 172.21.15.98:0/3403836370 conn(0x7f1940076da0 msgr2=0x7f19400791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:35.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.547+0000 7f1945200640 1 -- 172.21.15.98:0/3403836370 shutdown_connections 2024-03-16T12:21:35.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.547+0000 7f1945200640 1 -- 172.21.15.98:0/3403836370 wait complete. 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 Processor -- start 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 -- start start 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 0x7f194019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 0x7f19401a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1940113a80 con 0x7f1940108ff0 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1940113900 con 0x7f19401099b0 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.548+0000 7f1945200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1940113c00 con 0x7f1940107e00 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 0x7f19401a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:45000/0 (socket says 172.21.15.98:45000) 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f1935400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 0x7f194019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 -- 172.21.15.98:0/3787699625 learned_addr learned my addr 172.21.15.98:0/3787699625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 -- 172.21.15.98:0/3787699625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 msgr2=0x7f19401a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 0x7f19401a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 -- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 msgr2=0x7f194019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 0x7f194019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 -- 172.21.15.98:0/3787699625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19401a5cc0 con 0x7f1940107e00 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193e800640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 0x7f19401a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f193de00640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9c0 tx=0x7f193000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000cd80 con 0x7f1940107e00 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300035f0 con 0x7f1940107e00 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19401a5fb0 con 0x7f1940107e00 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.549+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930010690 con 0x7f1940107e00 2024-03-16T12:21:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.550+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19401a63c0 con 0x7f1940107e00 2024-03-16T12:21:35.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.551+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194010f120 con 0x7f1940107e00 2024-03-16T12:21:35.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.552+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f19300037a0 con 0x7f1940107e00 2024-03-16T12:21:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.554+0000 7f1937400640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 0x7f19200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.554+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19300cf180 con 0x7f1940107e00 2024-03-16T12:21:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.554+0000 7f1935400640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 0x7f19200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.555+0000 7f1935400640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 0x7f19200a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f194019f5b0 tx=0x7f192800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.557+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f193009be60 con 0x7f1940107e00 2024-03-16T12:21:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.745+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19400624e0 con 0x7f1940107e00 2024-03-16T12:21:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.746+0000 7f1937400640 1 -- 172.21.15.98:0/3787699625 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f193009b760 con 0x7f1940107e00 2024-03-16T12:21:35.727 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 msgr2=0x7f19200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 0x7f19200a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f194019f5b0 tx=0x7f192800b410 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 msgr2=0x7f194019e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9c0 tx=0x7f193000ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 shutdown_connections 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f19200a2800 0x7f19200a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19401099b0 0x7f19401a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1940108ff0 0x7f194019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 --2- 172.21.15.98:0/3787699625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1940107e00 0x7f194019e0a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.748+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 >> 172.21.15.98:0/3787699625 conn(0x7f1940076da0 msgr2=0x7f194010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.749+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 shutdown_connections 2024-03-16T12:21:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.749+0000 7f1945200640 1 -- 172.21.15.98:0/3787699625 wait complete. 2024-03-16T12:21:35.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:21:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 -- 172.21.15.98:0/4176105947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 msgr2=0x7f6bb8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 --2- 172.21.15.98:0/4176105947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb8109470 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009900 tx=0x7f6ba002bc40 comp rx=0 tx=0).stop 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 -- 172.21.15.98:0/4176105947 shutdown_connections 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 --2- 172.21.15.98:0/4176105947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 --2- 172.21.15.98:0/4176105947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb8109470 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 --2- 172.21.15.98:0/4176105947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 0x7f6bb8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 -- 172.21.15.98:0/4176105947 >> 172.21.15.98:0/4176105947 conn(0x7f6bb8076da0 msgr2=0x7f6bb80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 -- 172.21.15.98:0/4176105947 shutdown_connections 2024-03-16T12:21:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.970+0000 7f6bbd200640 1 -- 172.21.15.98:0/4176105947 wait complete. 2024-03-16T12:21:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 Processor -- start 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 -- start start 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 0x7f6bb819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb81a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.971+0000 7f6bbd200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8113a80 con 0x7f6bb81099b0 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bbd200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8113900 con 0x7f6bb8108ff0 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bbd200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8113c00 con 0x7f6bb8107e00 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bb5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 0x7f6bb819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50708/0 (socket says 172.21.15.98:50708) 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 -- 172.21.15.98:0/3299105026 learned_addr learned my addr 172.21.15.98:0/3299105026 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bb6800640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb81a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 -- 172.21.15.98:0/3299105026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 msgr2=0x7f6bb819e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 0x7f6bb819e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 -- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 msgr2=0x7f6bb81a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb81a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 -- 172.21.15.98:0/3299105026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb81a5d10 con 0x7f6bb8108ff0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bb6800640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb81a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.972+0000 7f6bad400640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009230 tx=0x7f6ba002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.973+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0041070 con 0x7f6bb8108ff0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.973+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba0004450 con 0x7f6bb8108ff0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.973+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb81a5fa0 con 0x7f6bb8108ff0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.973+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0003e00 con 0x7f6bb8108ff0 2024-03-16T12:21:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.973+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb81a6400 con 0x7f6bb8108ff0 2024-03-16T12:21:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.974+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb810f120 con 0x7f6bb8108ff0 2024-03-16T12:21:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.976+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ba0044070 con 0x7f6bb8108ff0 2024-03-16T12:21:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.977+0000 7f6baf400640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 0x7f6b9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.977+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6ba00f0ab0 con 0x7f6bb8108ff0 2024-03-16T12:21:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.977+0000 7f6bb5e00640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 0x7f6b9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:35.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.978+0000 7f6bb5e00640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 0x7f6b9c0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8108e50 tx=0x7f6ba8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:35.984+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ba00bd790 con 0x7f6bb8108ff0 2024-03-16T12:21:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.171+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6bb802d1b0 con 0x7f6bb8108ff0 2024-03-16T12:21:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.171+0000 7f6baf400640 1 -- 172.21.15.98:0/3299105026 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f6ba00bd090 con 0x7f6bb8108ff0 2024-03-16T12:21:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 msgr2=0x7f6b9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 0x7f6b9c0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8108e50 tx=0x7f6ba8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 msgr2=0x7f6bb819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009230 tx=0x7f6ba002c660 comp rx=0 tx=0).stop 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 shutdown_connections 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6b9c0a2880 0x7f6b9c0a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb81099b0 0x7f6bb81a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bb8108ff0 0x7f6bb819e630 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 --2- 172.21.15.98:0/3299105026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8107e00 0x7f6bb819e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 >> 172.21.15.98:0/3299105026 conn(0x7f6bb8076da0 msgr2=0x7f6bb810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 shutdown_connections 2024-03-16T12:21:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.174+0000 7f6bbd200640 1 -- 172.21.15.98:0/3299105026 wait complete. 2024-03-16T12:21:36.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:35 smithi098 ceph-mon[31633]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T12:21:36.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:35 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1567894680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:35 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3787699625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:36.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:35 smithi153 ceph-mon[36901]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T12:21:36.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:35 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1567894680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:35 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3787699625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:35 smithi173 ceph-mon[36632]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T12:21:36.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:35 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1567894680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:35 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3787699625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 -- 172.21.15.98:0/1976531557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 msgr2=0x7f38b0108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 --2- 172.21.15.98:0/1976531557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b0108220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009900 tx=0x7f38a402bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 -- 172.21.15.98:0/1976531557 shutdown_connections 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 --2- 172.21.15.98:0/1976531557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b01099b0 0x7f38b0110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 --2- 172.21.15.98:0/1976531557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b0108ff0 0x7f38b0109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 --2- 172.21.15.98:0/1976531557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b0108220 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 -- 172.21.15.98:0/1976531557 >> 172.21.15.98:0/1976531557 conn(0x7f38b0076dc0 msgr2=0x7f38b00791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 -- 172.21.15.98:0/1976531557 shutdown_connections 2024-03-16T12:21:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.386+0000 7f38b8800640 1 -- 172.21.15.98:0/1976531557 wait complete. 2024-03-16T12:21:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.387+0000 7f38b8800640 1 Processor -- start 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.387+0000 7f38b8800640 1 -- start start 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 0x7f38b019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01099b0 0x7f38b01a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0113ad0 con 0x7f38b01099b0 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0113950 con 0x7f38b0108ff0 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b8800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0113c50 con 0x7f38b0107e00 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 0x7f38b019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 0x7f38b019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50726/0 (socket says 172.21.15.98:50726) 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5400640 1 -- 172.21.15.98:0/3530414022 learned_addr learned my addr 172.21.15.98:0/3530414022 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b6800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01099b0 0x7f38b01a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 -- 172.21.15.98:0/3530414022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 msgr2=0x7f38b019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 0x7f38b019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 -- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01099b0 msgr2=0x7f38b01a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01099b0 0x7f38b01a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.388+0000 7f38b5e00640 1 -- 172.21.15.98:0/3530414022 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b01a5d50 con 0x7f38b0107e00 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38b5e00640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b019e130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f38a4036040 tx=0x7f38a402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4043020 con 0x7f38b0107e00 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a40054e0 con 0x7f38b0107e00 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4003c30 con 0x7f38b0107e00 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b01a5fe0 con 0x7f38b0107e00 2024-03-16T12:21:36.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.389+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b01a6390 con 0x7f38b0107e00 2024-03-16T12:21:36.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.390+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b010f0c0 con 0x7f38b0107e00 2024-03-16T12:21:36.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.392+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f38a4046020 con 0x7f38b0107e00 2024-03-16T12:21:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.393+0000 7f38aaa00640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.393+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f38a40f0dd0 con 0x7f38b0107e00 2024-03-16T12:21:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.393+0000 7f38b5400640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.394+0000 7f38b5400640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 0x7f38980a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f38b019f640 tx=0x7f389c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.397+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38a404d020 con 0x7f38b0107e00 2024-03-16T12:21:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.591+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38b00624b0 con 0x7f38b0107e00 2024-03-16T12:21:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.593+0000 7f38aaa00640 1 -- 172.21.15.98:0/3530414022 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f38a40bdab0 con 0x7f38b0107e00 2024-03-16T12:21:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:36.575 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 msgr2=0x7f38980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 0x7f38980a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f38b019f640 tx=0x7f389c005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 msgr2=0x7f38b019e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b019e130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f38a4036040 tx=0x7f38a402c650 comp rx=0 tx=0).stop 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 shutdown_connections 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f38980a2800 0x7f38980a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38b01099b0 0x7f38b01a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b0108ff0 0x7f38b019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 --2- 172.21.15.98:0/3530414022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38b0107e00 0x7f38b019e130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 >> 172.21.15.98:0/3530414022 conn(0x7f38b0076dc0 msgr2=0x7f38b010c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 shutdown_connections 2024-03-16T12:21:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.596+0000 7f38b8800640 1 -- 172.21.15.98:0/3530414022 wait complete. 2024-03-16T12:21:36.595 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:21:36.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 -- 172.21.15.98:0/2685461237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 msgr2=0x7fd668109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 --2- 172.21.15.98:0/2685461237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd668109810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd660009ca0 tx=0x7fd66002c100 comp rx=0 tx=0).stop 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 -- 172.21.15.98:0/2685461237 shutdown_connections 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 --2- 172.21.15.98:0/2685461237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd668109810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 --2- 172.21.15.98:0/2685461237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668104a90 0x7fd668106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 --2- 172.21.15.98:0/2685461237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668102160 0x7fd668104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.820+0000 7fd66e400640 1 -- 172.21.15.98:0/2685461237 >> 172.21.15.98:0/2685461237 conn(0x7fd6680fbb50 msgr2=0x7fd6680fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.821+0000 7fd66e400640 1 -- 172.21.15.98:0/2685461237 shutdown_connections 2024-03-16T12:21:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.821+0000 7fd66e400640 1 -- 172.21.15.98:0/2685461237 wait complete. 2024-03-16T12:21:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 Processor -- start 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 -- start start 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 0x7fd668199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 0x7fd66819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd6681a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d360 con 0x7fd668104a90 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d1e0 con 0x7fd6681073c0 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66e400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d4e0 con 0x7fd668102160 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd667e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 0x7fd66819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66da00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd6681a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd667e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 0x7fd66819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55824/0 (socket says 172.21.15.98:55824) 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd66d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 0x7fd668199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.822+0000 7fd667e00640 1 -- 172.21.15.98:0/2401308623 learned_addr learned my addr 172.21.15.98:0/2401308623 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 -- 172.21.15.98:0/2401308623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 msgr2=0x7fd6681a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd6681a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 -- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 msgr2=0x7fd66819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 0x7fd66819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 -- 172.21.15.98:0/2401308623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6681a17f0 con 0x7fd668102160 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66d000640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 0x7fd668199bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00e9f0 tx=0x7fd65c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c00ccc0 con 0x7fd668102160 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c010910 con 0x7fd668102160 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6681a1ae0 con 0x7fd668102160 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c018960 con 0x7fd668102160 2024-03-16T12:21:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.823+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6681a2030 con 0x7fd668102160 2024-03-16T12:21:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.824+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd668108eb0 con 0x7fd668102160 2024-03-16T12:21:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.826+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd65c010ac0 con 0x7fd668102160 2024-03-16T12:21:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.827+0000 7fd665600640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 0x7fd6500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.828+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd65c0cf1b0 con 0x7fd668102160 2024-03-16T12:21:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.828+0000 7fd667e00640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 0x7fd6500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.828+0000 7fd667e00640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 0x7fd6500a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd6540095f0 tx=0x7fd654005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:36.830+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd65c09b840 con 0x7fd668102160 2024-03-16T12:21:37.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.029+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd66802d2d0 con 0x7fd668102160 2024-03-16T12:21:37.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.029+0000 7fd665600640 1 -- 172.21.15.98:0/2401308623 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fd65c09f8e0 con 0x7fd668102160 2024-03-16T12:21:37.011 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:21:37.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 msgr2=0x7fd6500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 0x7fd6500a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd6540095f0 tx=0x7fd654005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 msgr2=0x7fd668199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 0x7fd668199bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00e9f0 tx=0x7fd65c00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 shutdown_connections 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd6500a2800 0x7fd6500a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6681073c0 0x7fd6681a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd668104a90 0x7fd66819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 --2- 172.21.15.98:0/2401308623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd668102160 0x7fd668199bd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 >> 172.21.15.98:0/2401308623 conn(0x7fd6680fbb50 msgr2=0x7fd668101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 shutdown_connections 2024-03-16T12:21:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.032+0000 7fd66e400640 1 -- 172.21.15.98:0/2401308623 wait complete. 2024-03-16T12:21:37.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 -- 172.21.15.98:0/972029744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc104bf0 msgr2=0x7f97cc105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 --2- 172.21.15.98:0/972029744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc104bf0 0x7f97cc105070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009860 tx=0x7f97b802bb70 comp rx=0 tx=0).stop 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 -- 172.21.15.98:0/972029744 shutdown_connections 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 --2- 172.21.15.98:0/972029744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc1055b0 0x7f97cc10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 --2- 172.21.15.98:0/972029744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc104bf0 0x7f97cc105070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 --2- 172.21.15.98:0/972029744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 -- 172.21.15.98:0/972029744 >> 172.21.15.98:0/972029744 conn(0x7f97cc0fef90 msgr2=0x7f97cc1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:37.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 -- 172.21.15.98:0/972029744 shutdown_connections 2024-03-16T12:21:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.251+0000 7f97d4200640 1 -- 172.21.15.98:0/972029744 wait complete. 2024-03-16T12:21:37.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 Processor -- start 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 -- start start 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 0x7f97cc19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 0x7f97cc1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10f660 con 0x7f97cc103a00 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10f4e0 con 0x7f97cc1055b0 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d4200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10f7e0 con 0x7f97cc104bf0 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d2200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 0x7f97cc1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 0x7f97cc19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43020/0 (socket says 172.21.15.98:43020) 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d1800640 1 -- 172.21.15.98:0/734917112 learned_addr learned my addr 172.21.15.98:0/734917112 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:37.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.253+0000 7f97d0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 0x7f97cc19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56122/0 (socket says 172.21.15.98:56122) 2024-03-16T12:21:37.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 -- 172.21.15.98:0/734917112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 msgr2=0x7f97cc19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 0x7f97cc19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 -- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 msgr2=0x7f97cc199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 -- 172.21.15.98:0/734917112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc1a1930 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d2200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 0x7f97cc1a1290 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ef00 tx=0x7f97c400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004b20 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c4003e50 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c400b570 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.254+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1a1c20 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.255+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc1a2120 con 0x7f97cc1055b0 2024-03-16T12:21:37.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.256+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97cc10ad20 con 0x7f97cc1055b0 2024-03-16T12:21:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.258+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f97c401d020 con 0x7f97cc1055b0 2024-03-16T12:21:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.259+0000 7f97ca000640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 0x7f97b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.259+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f97c40cef00 con 0x7f97cc1055b0 2024-03-16T12:21:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.259+0000 7f97d1800640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 0x7f97b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.260+0000 7f97d1800640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 0x7f97b40a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f97cc19b1f0 tx=0x7f97c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:37.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:36 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3299105026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:37.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:36 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3530414022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.266+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97c409bc40 con 0x7f97cc1055b0 2024-03-16T12:21:37.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:36 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3299105026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:37.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:36 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3530414022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:37.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3299105026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:37.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:36 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3530414022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.463+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97cc0623e0 con 0x7f97cc1055b0 2024-03-16T12:21:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.464+0000 7f97ca000640 1 -- 172.21.15.98:0/734917112 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f97c409b540 con 0x7f97cc1055b0 2024-03-16T12:21:37.445 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.466+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 msgr2=0x7f97b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.466+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 0x7f97b40a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f97cc19b1f0 tx=0x7f97c0005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.466+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 msgr2=0x7f97cc1a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.466+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 0x7f97cc1a1290 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ef00 tx=0x7f97c400c740 comp rx=0 tx=0).stop 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 shutdown_connections 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f97b40a2800 0x7f97b40a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97cc1055b0 0x7f97cc1a1290 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97cc104bf0 0x7f97cc19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 --2- 172.21.15.98:0/734917112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97cc103a00 0x7f97cc199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 >> 172.21.15.98:0/734917112 conn(0x7f97cc0fef90 msgr2=0x7f97cc107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 shutdown_connections 2024-03-16T12:21:37.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.467+0000 7f97d4200640 1 -- 172.21.15.98:0/734917112 wait complete. 2024-03-16T12:21:37.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:21:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 -- 172.21.15.98:0/2956351248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00773e0 msgr2=0x7f6cc0077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2956351248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00773e0 0x7f6cc0077840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009900 tx=0x7f6ca802bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 -- 172.21.15.98:0/2956351248 shutdown_connections 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2956351248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc0077d80 0x7f6cc0078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2956351248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00773e0 0x7f6cc0077840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2956351248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc00761a0 0x7f6cc00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 -- 172.21.15.98:0/2956351248 >> 172.21.15.98:0/2956351248 conn(0x7f6cc00fef90 msgr2=0x7f6cc01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 -- 172.21.15.98:0/2956351248 shutdown_connections 2024-03-16T12:21:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.705+0000 7f6cc6800640 1 -- 172.21.15.98:0/2956351248 wait complete. 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.706+0000 7f6cc6800640 1 Processor -- start 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.706+0000 7f6cc6800640 1 -- start start 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.706+0000 7f6cc6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 0x7f6cc019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.706+0000 7f6cc6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 0x7f6cc019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cc6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cc6800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0113af0 con 0x7f6cc00773e0 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cc6800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0113970 con 0x7f6cc0077d80 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cc6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0113c70 con 0x7f6cc00761a0 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 0x7f6cc019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54010/0 (socket says 172.21.15.98:54010) 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 -- 172.21.15.98:0/2002847944 learned_addr learned my addr 172.21.15.98:0/2002847944 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cb6a00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 0x7f6cc019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 -- 172.21.15.98:0/2002847944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 msgr2=0x7f6cc019e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 0x7f6cc019e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 -- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 msgr2=0x7f6cc019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 0x7f6cc019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 -- 172.21.15.98:0/2002847944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc01a5d40 con 0x7f6cc0077d80 2024-03-16T12:21:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbf400640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 0x7f6cc019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cb6a00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 0x7f6cc019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.707+0000 7f6cbfe00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009fd0 tx=0x7f6cb800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.708+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8017070 con 0x7f6cc0077d80 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.708+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb8003e50 con 0x7f6cc0077d80 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.708+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80106c0 con 0x7f6cc0077d80 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.708+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc01a5fd0 con 0x7f6cc0077d80 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.708+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc01a64f0 con 0x7f6cc0077d80 2024-03-16T12:21:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.709+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc0110390 con 0x7f6cc0077d80 2024-03-16T12:21:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.711+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6cb8003970 con 0x7f6cc0077d80 2024-03-16T12:21:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.712+0000 7f6cbcc00640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 0x7f6ca40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.712+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6cb80ceb70 con 0x7f6cc0077d80 2024-03-16T12:21:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.712+0000 7f6cbf400640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 0x7f6ca40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.713+0000 7f6cbf400640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 0x7f6ca40a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009fc0 tx=0x7f6cb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.716+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cb809b850 con 0x7f6cc0077d80 2024-03-16T12:21:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.919+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6cc00623e0 con 0x7f6cc0077d80 2024-03-16T12:21:37.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.920+0000 7f6cbcc00640 1 -- 172.21.15.98:0/2002847944 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f6cb809b150 con 0x7f6cc0077d80 2024-03-16T12:21:37.902 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:21:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 msgr2=0x7f6ca40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 0x7f6ca40a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009fc0 tx=0x7f6cb0005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 msgr2=0x7f6cc01a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009fd0 tx=0x7f6cb800b260 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 shutdown_connections 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6ca40a2880 0x7f6ca40a4d40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc0077d80 0x7f6cc01a56d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc00773e0 0x7f6cc019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 --2- 172.21.15.98:0/2002847944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cc00761a0 0x7f6cc019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 >> 172.21.15.98:0/2002847944 conn(0x7f6cc00fef90 msgr2=0x7f6cc0100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 shutdown_connections 2024-03-16T12:21:37.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:37.923+0000 7f6cc6800640 1 -- 172.21.15.98:0/2002847944 wait complete. 2024-03-16T12:21:37.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 -- 172.21.15.98:0/1009355273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00741c0 msgr2=0x7f58a0074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 --2- 172.21.15.98:0/1009355273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00741c0 0x7f58a0074620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5888009860 tx=0x7f588802bc00 comp rx=0 tx=0).stop 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 -- 172.21.15.98:0/1009355273 shutdown_connections 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 --2- 172.21.15.98:0/1009355273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a01096a0 0x7f58a010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 --2- 172.21.15.98:0/1009355273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00741c0 0x7f58a0074620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 --2- 172.21.15.98:0/1009355273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a00757a0 0x7f58a0073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 -- 172.21.15.98:0/1009355273 >> 172.21.15.98:0/1009355273 conn(0x7f58a00fcd80 msgr2=0x7f58a00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 -- 172.21.15.98:0/1009355273 shutdown_connections 2024-03-16T12:21:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.152+0000 7f58a6a00640 1 -- 172.21.15.98:0/1009355273 wait complete. 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 Processor -- start 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 -- start start 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 0x7f58a019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a01096a0 0x7f58a01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58a010f680 con 0x7f58a01096a0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58a010f500 con 0x7f58a00741c0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f58a6a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f58a010f800 con 0x7f58a00757a0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.154+0000 7f589fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 0x7f58a019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54034/0 (socket says 172.21.15.98:54034) 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 -- 172.21.15.98:0/3668691285 learned_addr learned my addr 172.21.15.98:0/3668691285 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f58a4a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a01096a0 0x7f58a01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 -- 172.21.15.98:0/3668691285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 msgr2=0x7f58a019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 0x7f58a019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 -- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a01096a0 msgr2=0x7f58a01a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a01096a0 0x7f58a01a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 -- 172.21.15.98:0/3668691285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a01a1940 con 0x7f58a00741c0 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589f400640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 0x7f58a019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589fe00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f589000ebc0 tx=0x7f589000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890010430 con 0x7f58a00741c0 2024-03-16T12:21:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5890010ab0 con 0x7f58a00741c0 2024-03-16T12:21:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.155+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890018b30 con 0x7f58a00741c0 2024-03-16T12:21:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.156+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a01a1c30 con 0x7f58a00741c0 2024-03-16T12:21:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.156+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a01a2130 con 0x7f58a00741c0 2024-03-16T12:21:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.157+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a0108460 con 0x7f58a00741c0 2024-03-16T12:21:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.158+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5890018cd0 con 0x7f58a00741c0 2024-03-16T12:21:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.160+0000 7f589cc00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 0x7f58840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.160+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58900cf550 con 0x7f58a00741c0 2024-03-16T12:21:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.160+0000 7f589f400640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 0x7f58840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.161+0000 7f589f400640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 0x7f58840a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5888009830 tx=0x7f5888034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.163+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f589009c230 con 0x7f58a00741c0 2024-03-16T12:21:38.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:37 smithi098 ceph-mon[31633]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:38.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2401308623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/734917112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:37 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2002847944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:37 smithi153 ceph-mon[36901]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:38.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2401308623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/734917112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:37 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2002847944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:37 smithi173 ceph-mon[36632]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:38.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2401308623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/734917112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:37 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2002847944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:38.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.362+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f58a0062480 con 0x7f58a00741c0 2024-03-16T12:21:38.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.363+0000 7f589cc00640 1 -- 172.21.15.98:0/3668691285 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f589009bb30 con 0x7f58a00741c0 2024-03-16T12:21:38.344 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 msgr2=0x7f58840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 0x7f58840a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5888009830 tx=0x7f5888034b50 comp rx=0 tx=0).stop 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 msgr2=0x7f58a0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f589000ebc0 tx=0x7f589000c2d0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 shutdown_connections 2024-03-16T12:21:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f58840a2800 0x7f58840a4cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a01096a0 0x7f58a01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f58a00757a0 0x7f58a019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 --2- 172.21.15.98:0/3668691285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a00741c0 0x7f58a0199d30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 >> 172.21.15.98:0/3668691285 conn(0x7f58a00fcd80 msgr2=0x7f58a0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 shutdown_connections 2024-03-16T12:21:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.366+0000 7f58a6a00640 1 -- 172.21.15.98:0/3668691285 wait complete. 2024-03-16T12:21:38.364 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-16T12:21:38.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 -- 172.21.15.98:0/550261241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 msgr2=0x7f3dc8074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 --2- 172.21.15.98:0/550261241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8074640 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009900 tx=0x7f3db002bcd0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 -- 172.21.15.98:0/550261241 shutdown_connections 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 --2- 172.21.15.98:0/550261241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 0x7f3dc810bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 --2- 172.21.15.98:0/550261241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8074640 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 --2- 172.21.15.98:0/550261241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 -- 172.21.15.98:0/550261241 >> 172.21.15.98:0/550261241 conn(0x7f3dc80fcd80 msgr2=0x7f3dc80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 -- 172.21.15.98:0/550261241 shutdown_connections 2024-03-16T12:21:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.592+0000 7f3dcd800640 1 -- 172.21.15.98:0/550261241 wait complete. 2024-03-16T12:21:38.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.593+0000 7f3dcd800640 1 Processor -- start 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.593+0000 7f3dcd800640 1 -- start start 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 0x7f3dc81a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810f650 con 0x7f3dc8109680 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810f4d0 con 0x7f3dc80741e0 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dcd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810f7d0 con 0x7f3dc80757c0 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dc7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 0x7f3dc81a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56176/0 (socket says 172.21.15.98:56176) 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 -- 172.21.15.98:0/822617418 learned_addr learned my addr 172.21.15.98:0/822617418 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:38.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dc6a00640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 -- 172.21.15.98:0/822617418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 msgr2=0x7f3dc8199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 -- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 msgr2=0x7f3dc81a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 0x7f3dc81a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.594+0000 7f3dbe000640 1 -- 172.21.15.98:0/822617418 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc81a1900 con 0x7f3dc80757c0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dbe000640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005ab0 tx=0x7f3db002c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0041070 con 0x7f3dc80757c0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3db0004450 con 0x7f3dc80757c0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc81a1b90 con 0x7f3dc80757c0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0034440 con 0x7f3dc80757c0 2024-03-16T12:21:38.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.595+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc81a2080 con 0x7f3dc80757c0 2024-03-16T12:21:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.596+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc8108440 con 0x7f3dc80757c0 2024-03-16T12:21:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.598+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3db0044070 con 0x7f3dc80757c0 2024-03-16T12:21:38.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.599+0000 7f3dbfe00640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 0x7f3dac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:38.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.599+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3db00f08a0 con 0x7f3dc80757c0 2024-03-16T12:21:38.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.600+0000 7f3dc6a00640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 0x7f3dac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:38.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.600+0000 7f3dc6a00640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 0x7f3dac0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3dc819ab90 tx=0x7f3db8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:38.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.603+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3db00bcfc0 con 0x7f3dc80757c0 2024-03-16T12:21:38.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.843+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dc8062480 con 0x7f3dc80757c0 2024-03-16T12:21:38.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.844+0000 7f3dbfe00640 1 -- 172.21.15.98:0/822617418 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7f3db00c1060 con 0x7f3dc80757c0 2024-03-16T12:21:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:38.826 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:21:38.826 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 msgr2=0x7f3dac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 0x7f3dac0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3dc819ab90 tx=0x7f3db8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 msgr2=0x7f3dc819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005ab0 tx=0x7f3db002c6e0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 shutdown_connections 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dac0a2800 0x7f3dac0a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc8109680 0x7f3dc81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dc80757c0 0x7f3dc819a200 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 --2- 172.21.15.98:0/822617418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dc80741e0 0x7f3dc8199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:38.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.847+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 >> 172.21.15.98:0/822617418 conn(0x7f3dc80fcd80 msgr2=0x7f3dc8105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.848+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 shutdown_connections 2024-03-16T12:21:38.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:38.848+0000 7f3dcd800640 1 -- 172.21.15.98:0/822617418 wait complete. 2024-03-16T12:21:38.848 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-16T12:21:38.848 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-16T12:21:38.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T12:21:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 -- 172.21.15.98:0/1221356909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 msgr2=0x7f0d24110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 --2- 172.21.15.98:0/1221356909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d24110240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c009ca0 tx=0x7f0d1c02c100 comp rx=0 tx=0).stop 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 -- 172.21.15.98:0/1221356909 shutdown_connections 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 --2- 172.21.15.98:0/1221356909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d24110240 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 --2- 172.21.15.98:0/1221356909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24108ff0 0x7f0d24109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 --2- 172.21.15.98:0/1221356909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24107e00 0x7f0d24108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 -- 172.21.15.98:0/1221356909 >> 172.21.15.98:0/1221356909 conn(0x7f0d24076dc0 msgr2=0x7f0d240791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 -- 172.21.15.98:0/1221356909 shutdown_connections 2024-03-16T12:21:39.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.078+0000 7f0d29e00640 1 -- 172.21.15.98:0/1221356909 wait complete. 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.079+0000 7f0d29e00640 1 Processor -- start 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 -- start start 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24107e00 0x7f0d2419e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d241a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24113a60 con 0x7f0d24107e00 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d241138e0 con 0x7f0d241099b0 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d29e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24113be0 con 0x7f0d24108ff0 2024-03-16T12:21:39.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d1be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d22a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24107e00 0x7f0d2419e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d23400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d241a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d1be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56184/0 (socket says 172.21.15.98:56184) 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.080+0000 7f0d1be00640 1 -- 172.21.15.98:0/4261615926 learned_addr learned my addr 172.21.15.98:0/4261615926 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 -- 172.21.15.98:0/4261615926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 msgr2=0x7f0d241a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d241a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 -- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24107e00 msgr2=0x7f0d2419e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24107e00 0x7f0d2419e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 -- 172.21.15.98:0/4261615926 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d241a5c60 con 0x7f0d24108ff0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d23400640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d241a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d1be00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9f0 tx=0x7f0d0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00cd80 con 0x7f0d24108ff0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d0c010910 con 0x7f0d24108ff0 2024-03-16T12:21:39.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c018990 con 0x7f0d24108ff0 2024-03-16T12:21:39.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d241a5f50 con 0x7f0d24108ff0 2024-03-16T12:21:39.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.081+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d241a63c0 con 0x7f0d24108ff0 2024-03-16T12:21:39.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.083+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2410f0c0 con 0x7f0d24108ff0 2024-03-16T12:21:39.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.084+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0d0c010ac0 con 0x7f0d24108ff0 2024-03-16T12:21:39.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.085+0000 7f0d28a00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 0x7f0d080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.086+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0d0c0cf150 con 0x7f0d24108ff0 2024-03-16T12:21:39.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.086+0000 7f0d22a00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 0x7f0d080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.086+0000 7f0d22a00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 0x7f0d080a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0d24108e50 tx=0x7f0d1400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:39.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.089+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d0c09b7e0 con 0x7f0d24108ff0 2024-03-16T12:21:39.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:38 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3668691285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:39.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:38 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/822617418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:39.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:38 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3668691285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:39.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:38 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/822617418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.287+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d24061c20 con 0x7f0d080a2800 2024-03-16T12:21:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.304+0000 7f0d28a00640 1 -- 172.21.15.98:0/4261615926 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340494 (secure 0 0 0) 0x7f0d24061c20 con 0x7f0d080a2800 2024-03-16T12:21:39.291 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:21:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 msgr2=0x7f0d080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 0x7f0d080a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0d24108e50 tx=0x7f0d1400b390 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 msgr2=0x7f0d2419e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9f0 tx=0x7f0d0c00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 shutdown_connections 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0d080a2800 0x7f0d080a4cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d241099b0 0x7f0d241a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d24108ff0 0x7f0d2419e580 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 --2- 172.21.15.98:0/4261615926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24107e00 0x7f0d2419e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 >> 172.21.15.98:0/4261615926 conn(0x7f0d24076dc0 msgr2=0x7f0d2410c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 shutdown_connections 2024-03-16T12:21:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.312+0000 7f0d29e00640 1 -- 172.21.15.98:0/4261615926 wait complete. 2024-03-16T12:21:39.313 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-16T12:21:38.851289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662787,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1055,"num_read_kb":650,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384060,"kb_used_data":14108,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557764,"statfs":{"total":1151940427776,"available":1151547150336,"internally_reserved":0,"allocated":14446592,"data_stored":7974071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5993,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":43,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004627"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803803+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:20:12.803803+0000","last_peered":"2024-03-16T12:20:12.803803+0000","last_clean":"2024-03-16T12:20:12.803803+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:20:12.803803+0000","last_undegraded":"2024-03-16T12:20:12.803803+0000","last_fullsized":"2024-03-16T12:20:12.803803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798435+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:20:12.798435+0000","last_peered":"2024-03-16T12:20:12.798435+0000","last_clean":"2024-03-16T12:20:12.798435+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:20:12.798435+0000","last_undegraded":"2024-03-16T12:20:12.798435+0000","last_fullsized":"2024-03-16T12:20:12.798435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800413+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:20:12.800413+0000","last_peered":"2024-03-16T12:20:12.800413+0000","last_clean":"2024-03-16T12:20:12.800413+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:20:12.800413+0000","last_undegraded":"2024-03-16T12:20:12.800413+0000","last_fullsized":"2024-03-16T12:20:12.800413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755748+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:20:12.755748+0000","last_peered":"2024-03-16T12:20:12.755748+0000","last_clean":"2024-03-16T12:20:12.755748+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:20:12.755748+0000","last_undegraded":"2024-03-16T12:20:12.755748+0000","last_fullsized":"2024-03-16T12:20:12.755748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791944+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:20:12.791944+0000","last_peered":"2024-03-16T12:20:12.791944+0000","last_clean":"2024-03-16T12:20:12.791944+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:20:12.791944+0000","last_undegraded":"2024-03-16T12:20:12.791944+0000","last_fullsized":"2024-03-16T12:20:12.791944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785928+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:20:12.785928+0000","last_peered":"2024-03-16T12:20:12.785928+0000","last_clean":"2024-03-16T12:20:12.785928+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:20:12.785928+0000","last_undegraded":"2024-03-16T12:20:12.785928+0000","last_fullsized":"2024-03-16T12:20:12.785928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.773384+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:20:12.773384+0000","last_peered":"2024-03-16T12:20:12.773384+0000","last_clean":"2024-03-16T12:20:12.773384+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:20:12.773384+0000","last_undegraded":"2024-03-16T12:20:12.773384+0000","last_fullsized":"2024-03-16T12:20:12.773384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793133+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:20:12.793133+0000","last_peered":"2024-03-16T12:20:12.793133+0000","last_clean":"2024-03-16T12:20:12.793133+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:20:12.793133+0000","last_undegraded":"2024-03-16T12:20:12.793133+0000","last_fullsized":"2024-03-16T12:20:12.793133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785095+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:20:12.785095+0000","last_peered":"2024-03-16T12:20:12.785095+0000","last_clean":"2024-03-16T12:20:12.785095+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:20:12.785095+0000","last_undegraded":"2024-03-16T12:20:12.785095+0000","last_fullsized":"2024-03-16T12:20:12.785095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785364+0000","last_change":"2024-03-16T12:19:44.848697+0000","last_active":"2024-03-16T12:20:12.785364+0000","last_peered":"2024-03-16T12:20:12.785364+0000","last_clean":"2024-03-16T12:20:12.785364+0000","last_became_active":"2024-03-16T12:19:44.848414+0000","last_became_peered":"2024-03-16T12:19:44.848414+0000","last_unstale":"2024-03-16T12:20:12.785364+0000","last_undegraded":"2024-03-16T12:20:12.785364+0000","last_fullsized":"2024-03-16T12:20:12.785364+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:54.721056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786403+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:20:12.786403+0000","last_peered":"2024-03-16T12:20:12.786403+0000","last_clean":"2024-03-16T12:20:12.786403+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:20:12.786403+0000","last_undegraded":"2024-03-16T12:20:12.786403+0000","last_fullsized":"2024-03-16T12:20:12.786403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755018+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:20:12.755018+0000","last_peered":"2024-03-16T12:20:12.755018+0000","last_clean":"2024-03-16T12:20:12.755018+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:20:12.755018+0000","last_undegraded":"2024-03-16T12:20:12.755018+0000","last_fullsized":"2024-03-16T12:20:12.755018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790423+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:20:12.790423+0000","last_peered":"2024-03-16T12:20:12.790423+0000","last_clean":"2024-03-16T12:20:12.790423+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:20:12.790423+0000","last_undegraded":"2024-03-16T12:20:12.790423+0000","last_fullsized":"2024-03-16T12:20:12.790423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794049+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:20:12.794049+0000","last_peered":"2024-03-16T12:20:12.794049+0000","last_clean":"2024-03-16T12:20:12.794049+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:20:12.794049+0000","last_undegraded":"2024-03-16T12:20:12.794049+0000","last_fullsized":"2024-03-16T12:20:12.794049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.795950+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:20:12.795950+0000","last_peered":"2024-03-16T12:20:12.795950+0000","last_clean":"2024-03-16T12:20:12.795950+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:20:12.795950+0000","last_undegraded":"2024-03-16T12:20:12.795950+0000","last_fullsized":"2024-03-16T12:20:12.795950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803918+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:20:12.803918+0000","last_peered":"2024-03-16T12:20:12.803918+0000","last_clean":"2024-03-16T12:20:12.803918+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:20:12.803918+0000","last_undegraded":"2024-03-16T12:20:12.803918+0000","last_fullsized":"2024-03-16T12:20:12.803918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804028+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:20:12.804028+0000","last_peered":"2024-03-16T12:20:12.804028+0000","last_clean":"2024-03-16T12:20:12.804028+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:20:12.804028+0000","last_undegraded":"2024-03-16T12:20:12.804028+0000","last_fullsized":"2024-03-16T12:20:12.804028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786205+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:20:12.786205+0000","last_peered":"2024-03-16T12:20:12.786205+0000","last_clean":"2024-03-16T12:20:12.786205+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:20:12.786205+0000","last_undegraded":"2024-03-16T12:20:12.786205+0000","last_fullsized":"2024-03-16T12:20:12.786205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792899+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:20:12.792899+0000","last_peered":"2024-03-16T12:20:12.792899+0000","last_clean":"2024-03-16T12:20:12.792899+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:20:12.792899+0000","last_undegraded":"2024-03-16T12:20:12.792899+0000","last_fullsized":"2024-03-16T12:20:12.792899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547296+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:19:44.547296+0000","last_peered":"2024-03-16T12:19:44.547296+0000","last_clean":"2024-03-16T12:19:44.547296+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:19:44.547296+0000","last_undegraded":"2024-03-16T12:19:44.547296+0000","last_fullsized":"2024-03-16T12:19:44.547296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847301+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:44.847301+0000","last_peered":"2024-03-16T12:19:44.847301+0000","last_clean":"2024-03-16T12:19:44.847301+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:44.847301+0000","last_undegraded":"2024-03-16T12:19:44.847301+0000","last_fullsized":"2024-03-16T12:19:44.847301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.769143+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:20:12.769143+0000","last_peered":"2024-03-16T12:20:12.769143+0000","last_clean":"2024-03-16T12:20:12.769143+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:20:12.769143+0000","last_undegraded":"2024-03-16T12:20:12.769143+0000","last_fullsized":"2024-03-16T12:20:12.769143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559668+0000","last_change":"2024-03-16T12:19:44.559192+0000","last_active":"2024-03-16T12:19:44.559668+0000","last_peered":"2024-03-16T12:19:44.559668+0000","last_clean":"2024-03-16T12:19:44.559668+0000","last_became_active":"2024-03-16T12:19:44.558795+0000","last_became_peered":"2024-03-16T12:19:44.558795+0000","last_unstale":"2024-03-16T12:19:44.559668+0000","last_undegraded":"2024-03-16T12:19:44.559668+0000","last_fullsized":"2024-03-16T12:19:44.559668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:41:28.267737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.552034+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:19:44.552034+0000","last_peered":"2024-03-16T12:19:44.552034+0000","last_clean":"2024-03-16T12:19:44.552034+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:19:44.552034+0000","last_undegraded":"2024-03-16T12:19:44.552034+0000","last_fullsized":"2024-03-16T12:19:44.552034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804064+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:20:12.804064+0000","last_peered":"2024-03-16T12:20:12.804064+0000","last_clean":"2024-03-16T12:20:12.804064+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:20:12.804064+0000","last_undegraded":"2024-03-16T12:20:12.804064+0000","last_fullsized":"2024-03-16T12:20:12.804064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546380+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:19:44.546380+0000","last_peered":"2024-03-16T12:19:44.546380+0000","last_clean":"2024-03-16T12:19:44.546380+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:19:44.546380+0000","last_undegraded":"2024-03-16T12:19:44.546380+0000","last_fullsized":"2024-03-16T12:19:44.546380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543851+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:44.543851+0000","last_peered":"2024-03-16T12:19:44.543851+0000","last_clean":"2024-03-16T12:19:44.543851+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:44.543851+0000","last_undegraded":"2024-03-16T12:19:44.543851+0000","last_fullsized":"2024-03-16T12:19:44.543851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789084+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:20:12.789084+0000","last_peered":"2024-03-16T12:20:12.789084+0000","last_clean":"2024-03-16T12:20:12.789084+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:20:12.789084+0000","last_undegraded":"2024-03-16T12:20:12.789084+0000","last_fullsized":"2024-03-16T12:20:12.789084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546007+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:19:44.546007+0000","last_peered":"2024-03-16T12:19:44.546007+0000","last_clean":"2024-03-16T12:19:44.546007+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:19:44.546007+0000","last_undegraded":"2024-03-16T12:19:44.546007+0000","last_fullsized":"2024-03-16T12:19:44.546007+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543938+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:19:44.543938+0000","last_peered":"2024-03-16T12:19:44.543938+0000","last_clean":"2024-03-16T12:19:44.543938+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:19:44.543938+0000","last_undegraded":"2024-03-16T12:19:44.543938+0000","last_fullsized":"2024-03-16T12:19:44.543938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.784812+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:20:12.784812+0000","last_peered":"2024-03-16T12:20:12.784812+0000","last_clean":"2024-03-16T12:20:12.784812+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:20:12.784812+0000","last_undegraded":"2024-03-16T12:20:12.784812+0000","last_fullsized":"2024-03-16T12:20:12.784812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547794+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:19:44.547794+0000","last_peered":"2024-03-16T12:19:44.547794+0000","last_clean":"2024-03-16T12:19:44.547794+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:19:44.547794+0000","last_undegraded":"2024-03-16T12:19:44.547794+0000","last_fullsized":"2024-03-16T12:19:44.547794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":187,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.644775+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:21:37.644775+0000","last_peered":"2024-03-16T12:21:37.644775+0000","last_clean":"2024-03-16T12:21:37.644775+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:21:37.644775+0000","last_undegraded":"2024-03-16T12:21:37.644775+0000","last_fullsized":"2024-03-16T12:21:37.644775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781031+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:20:12.781031+0000","last_peered":"2024-03-16T12:20:12.781031+0000","last_clean":"2024-03-16T12:20:12.781031+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:20:12.781031+0000","last_undegraded":"2024-03-16T12:20:12.781031+0000","last_fullsized":"2024-03-16T12:20:12.781031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548011+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:19:44.548011+0000","last_peered":"2024-03-16T12:19:44.548011+0000","last_clean":"2024-03-16T12:19:44.548011+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:19:44.548011+0000","last_undegraded":"2024-03-16T12:19:44.548011+0000","last_fullsized":"2024-03-16T12:19:44.548011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547430+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:19:44.547430+0000","last_peered":"2024-03-16T12:19:44.547430+0000","last_clean":"2024-03-16T12:19:44.547430+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:19:44.547430+0000","last_undegraded":"2024-03-16T12:19:44.547430+0000","last_fullsized":"2024-03-16T12:19:44.547430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798746+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:20:12.798746+0000","last_peered":"2024-03-16T12:20:12.798746+0000","last_clean":"2024-03-16T12:20:12.798746+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:20:12.798746+0000","last_undegraded":"2024-03-16T12:20:12.798746+0000","last_fullsized":"2024-03-16T12:20:12.798746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545612+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:19:44.545612+0000","last_peered":"2024-03-16T12:19:44.545612+0000","last_clean":"2024-03-16T12:19:44.545612+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:19:44.545612+0000","last_undegraded":"2024-03-16T12:19:44.545612+0000","last_fullsized":"2024-03-16T12:19:44.545612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555984+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:44.555984+0000","last_peered":"2024-03-16T12:19:44.555984+0000","last_clean":"2024-03-16T12:19:44.555984+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:44.555984+0000","last_undegraded":"2024-03-16T12:19:44.555984+0000","last_fullsized":"2024-03-16T12:19:44.555984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792926+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:20:12.792926+0000","last_peered":"2024-03-16T12:20:12.792926+0000","last_clean":"2024-03-16T12:20:12.792926+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:20:12.792926+0000","last_undegraded":"2024-03-16T12:20:12.792926+0000","last_fullsized":"2024-03-16T12:20:12.792926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172396+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:19:45.172396+0000","last_peered":"2024-03-16T12:19:45.172396+0000","last_clean":"2024-03-16T12:19:45.172396+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:19:45.172396+0000","last_undegraded":"2024-03-16T12:19:45.172396+0000","last_fullsized":"2024-03-16T12:19:45.172396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549572+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:19:44.549572+0000","last_peered":"2024-03-16T12:19:44.549572+0000","last_clean":"2024-03-16T12:19:44.549572+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:19:44.549572+0000","last_undegraded":"2024-03-16T12:19:44.549572+0000","last_fullsized":"2024-03-16T12:19:44.549572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.801975+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:20:12.801975+0000","last_peered":"2024-03-16T12:20:12.801975+0000","last_clean":"2024-03-16T12:20:12.801975+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:20:12.801975+0000","last_undegraded":"2024-03-16T12:20:12.801975+0000","last_fullsized":"2024-03-16T12:20:12.801975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546523+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:19:44.546523+0000","last_peered":"2024-03-16T12:19:44.546523+0000","last_clean":"2024-03-16T12:19:44.546523+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:19:44.546523+0000","last_undegraded":"2024-03-16T12:19:44.546523+0000","last_fullsized":"2024-03-16T12:19:44.546523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":181,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:35.641987+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:21:35.641987+0000","last_peered":"2024-03-16T12:21:35.641987+0000","last_clean":"2024-03-16T12:21:35.641987+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:21:35.641987+0000","last_undegraded":"2024-03-16T12:21:35.641987+0000","last_fullsized":"2024-03-16T12:21:35.641987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787423+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:20:12.787423+0000","last_peered":"2024-03-16T12:20:12.787423+0000","last_clean":"2024-03-16T12:20:12.787423+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:20:12.787423+0000","last_undegraded":"2024-03-16T12:20:12.787423+0000","last_fullsized":"2024-03-16T12:20:12.787423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845673+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:19:44.845673+0000","last_peered":"2024-03-16T12:19:44.845673+0000","last_clean":"2024-03-16T12:19:44.845673+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:19:44.845673+0000","last_undegraded":"2024-03-16T12:19:44.845673+0000","last_fullsized":"2024-03-16T12:19:44.845673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548131+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:44.548131+0000","last_peered":"2024-03-16T12:19:44.548131+0000","last_clean":"2024-03-16T12:19:44.548131+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:44.548131+0000","last_undegraded":"2024-03-16T12:19:44.548131+0000","last_fullsized":"2024-03-16T12:19:44.548131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794150+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:20:12.794150+0000","last_peered":"2024-03-16T12:20:12.794150+0000","last_clean":"2024-03-16T12:20:12.794150+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:20:12.794150+0000","last_undegraded":"2024-03-16T12:20:12.794150+0000","last_fullsized":"2024-03-16T12:20:12.794150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546597+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:19:44.546597+0000","last_peered":"2024-03-16T12:19:44.546597+0000","last_clean":"2024-03-16T12:19:44.546597+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:19:44.546597+0000","last_undegraded":"2024-03-16T12:19:44.546597+0000","last_fullsized":"2024-03-16T12:19:44.546597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549488+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:44.549488+0000","last_peered":"2024-03-16T12:19:44.549488+0000","last_clean":"2024-03-16T12:19:44.549488+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:44.549488+0000","last_undegraded":"2024-03-16T12:19:44.549488+0000","last_fullsized":"2024-03-16T12:19:44.549488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785886+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:20:12.785886+0000","last_peered":"2024-03-16T12:20:12.785886+0000","last_clean":"2024-03-16T12:20:12.785886+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:20:12.785886+0000","last_undegraded":"2024-03-16T12:20:12.785886+0000","last_fullsized":"2024-03-16T12:20:12.785886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845685+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:19:44.845685+0000","last_peered":"2024-03-16T12:19:44.845685+0000","last_clean":"2024-03-16T12:19:44.845685+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:19:44.845685+0000","last_undegraded":"2024-03-16T12:19:44.845685+0000","last_fullsized":"2024-03-16T12:19:44.845685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'40","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:36.643509+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:21:36.643509+0000","last_peered":"2024-03-16T12:21:36.643509+0000","last_clean":"2024-03-16T12:21:36.643509+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:21:36.643509+0000","last_undegraded":"2024-03-16T12:21:36.643509+0000","last_fullsized":"2024-03-16T12:21:36.643509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68265,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":222,"num_write":41,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.753180+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:20:12.753180+0000","last_peered":"2024-03-16T12:20:12.753180+0000","last_clean":"2024-03-16T12:20:12.753180+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:20:12.753180+0000","last_undegraded":"2024-03-16T12:20:12.753180+0000","last_fullsized":"2024-03-16T12:20:12.753180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550225+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:19:44.550225+0000","last_peered":"2024-03-16T12:19:44.550225+0000","last_clean":"2024-03-16T12:19:44.550225+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:19:44.550225+0000","last_undegraded":"2024-03-16T12:19:44.550225+0000","last_fullsized":"2024-03-16T12:19:44.550225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565318+0000","last_change":"2024-03-16T12:19:44.565318+0000","last_active":"2024-03-16T12:19:44.565318+0000","last_peered":"2024-03-16T12:19:44.565318+0000","last_clean":"2024-03-16T12:19:44.565318+0000","last_became_active":"2024-03-16T12:19:44.551024+0000","last_became_peered":"2024-03-16T12:19:44.551024+0000","last_unstale":"2024-03-16T12:19:44.565318+0000","last_undegraded":"2024-03-16T12:19:44.565318+0000","last_fullsized":"2024-03-16T12:19:44.565318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:15.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800693+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:20:12.800693+0000","last_peered":"2024-03-16T12:20:12.800693+0000","last_clean":"2024-03-16T12:20:12.800693+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:20:12.800693+0000","last_undegraded":"2024-03-16T12:20:12.800693+0000","last_fullsized":"2024-03-16T12:20:12.800693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545017+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:19:44.545017+0000","last_peered":"2024-03-16T12:19:44.545017+0000","last_clean":"2024-03-16T12:19:44.545017+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:19:44.545017+0000","last_undegraded":"2024-03-16T12:19:44.545017+0000","last_fullsized":"2024-03-16T12:19:44.545017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549187+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:44.549187+0000","last_peered":"2024-03-16T12:19:44.549187+0000","last_clean":"2024-03-16T12:19:44.549187+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:44.549187+0000","last_undegraded":"2024-03-16T12:19:44.549187+0000","last_fullsized":"2024-03-16T12:19:44.549187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790351+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:20:12.790351+0000","last_peered":"2024-03-16T12:20:12.790351+0000","last_clean":"2024-03-16T12:20:12.790351+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:20:12.790351+0000","last_undegraded":"2024-03-16T12:20:12.790351+0000","last_fullsized":"2024-03-16T12:20:12.790351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.846970+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:19:44.846970+0000","last_peered":"2024-03-16T12:19:44.846970+0000","last_clean":"2024-03-16T12:19:44.846970+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:19:44.846970+0000","last_undegraded":"2024-03-16T12:19:44.846970+0000","last_fullsized":"2024-03-16T12:19:44.846970+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":182,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:32.719508+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:21:32.719508+0000","last_peered":"2024-03-16T12:21:32.719508+0000","last_clean":"2024-03-16T12:21:32.719508+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:21:32.719508+0000","last_undegraded":"2024-03-16T12:21:32.719508+0000","last_fullsized":"2024-03-16T12:21:32.719508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":898,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755629+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:20:12.755629+0000","last_peered":"2024-03-16T12:20:12.755629+0000","last_clean":"2024-03-16T12:20:12.755629+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:20:12.755629+0000","last_undegraded":"2024-03-16T12:20:12.755629+0000","last_fullsized":"2024-03-16T12:20:12.755629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567169+0000","last_change":"2024-03-16T12:19:44.565998+0000","last_active":"2024-03-16T12:19:44.567169+0000","last_peered":"2024-03-16T12:19:44.567169+0000","last_clean":"2024-03-16T12:19:44.567169+0000","last_became_active":"2024-03-16T12:19:44.564591+0000","last_became_peered":"2024-03-16T12:19:44.564591+0000","last_unstale":"2024-03-16T12:19:44.567169+0000","last_undegraded":"2024-03-16T12:19:44.567169+0000","last_fullsized":"2024-03-16T12:19:44.567169+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:57:48.227439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543368+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:44.543368+0000","last_peered":"2024-03-16T12:19:44.543368+0000","last_clean":"2024-03-16T12:19:44.543368+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:44.543368+0000","last_undegraded":"2024-03-16T12:19:44.543368+0000","last_fullsized":"2024-03-16T12:19:44.543368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782622+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:20:12.782622+0000","last_peered":"2024-03-16T12:20:12.782622+0000","last_clean":"2024-03-16T12:20:12.782622+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:20:12.782622+0000","last_undegraded":"2024-03-16T12:20:12.782622+0000","last_fullsized":"2024-03-16T12:20:12.782622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548932+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:19:44.548932+0000","last_peered":"2024-03-16T12:19:44.548932+0000","last_clean":"2024-03-16T12:19:44.548932+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:19:44.548932+0000","last_undegraded":"2024-03-16T12:19:44.548932+0000","last_fullsized":"2024-03-16T12:19:44.548932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543880+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:44.543880+0000","last_peered":"2024-03-16T12:19:44.543880+0000","last_clean":"2024-03-16T12:19:44.543880+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:44.543880+0000","last_undegraded":"2024-03-16T12:19:44.543880+0000","last_fullsized":"2024-03-16T12:19:44.543880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770720+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:20:12.770720+0000","last_peered":"2024-03-16T12:20:12.770720+0000","last_clean":"2024-03-16T12:20:12.770720+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:20:12.770720+0000","last_undegraded":"2024-03-16T12:20:12.770720+0000","last_fullsized":"2024-03-16T12:20:12.770720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543490+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:19:44.543490+0000","last_peered":"2024-03-16T12:19:44.543490+0000","last_clean":"2024-03-16T12:19:44.543490+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:19:44.543490+0000","last_undegraded":"2024-03-16T12:19:44.543490+0000","last_fullsized":"2024-03-16T12:19:44.543490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847452+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:19:44.847452+0000","last_peered":"2024-03-16T12:19:44.847452+0000","last_clean":"2024-03-16T12:19:44.847452+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:19:44.847452+0000","last_undegraded":"2024-03-16T12:19:44.847452+0000","last_fullsized":"2024-03-16T12:19:44.847452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793183+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:20:12.793183+0000","last_peered":"2024-03-16T12:20:12.793183+0000","last_clean":"2024-03-16T12:20:12.793183+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:20:12.793183+0000","last_undegraded":"2024-03-16T12:20:12.793183+0000","last_fullsized":"2024-03-16T12:20:12.793183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559770+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:19:44.559770+0000","last_peered":"2024-03-16T12:19:44.559770+0000","last_clean":"2024-03-16T12:19:44.559770+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:19:44.559770+0000","last_undegraded":"2024-03-16T12:19:44.559770+0000","last_fullsized":"2024-03-16T12:19:44.559770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547822+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:19:44.547822+0000","last_peered":"2024-03-16T12:19:44.547822+0000","last_clean":"2024-03-16T12:19:44.547822+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:19:44.547822+0000","last_undegraded":"2024-03-16T12:19:44.547822+0000","last_fullsized":"2024-03-16T12:19:44.547822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786284+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:20:12.786284+0000","last_peered":"2024-03-16T12:20:12.786284+0000","last_clean":"2024-03-16T12:20:12.786284+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:20:12.786284+0000","last_undegraded":"2024-03-16T12:20:12.786284+0000","last_fullsized":"2024-03-16T12:20:12.786284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559484+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:19:44.559484+0000","last_peered":"2024-03-16T12:19:44.559484+0000","last_clean":"2024-03-16T12:19:44.559484+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:19:44.559484+0000","last_undegraded":"2024-03-16T12:19:44.559484+0000","last_fullsized":"2024-03-16T12:19:44.559484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776444+0000","last_change":"2024-03-16T12:19:44.552241+0000","last_active":"2024-03-16T12:20:12.776444+0000","last_peered":"2024-03-16T12:20:12.776444+0000","last_clean":"2024-03-16T12:20:12.776444+0000","last_became_active":"2024-03-16T12:19:44.551762+0000","last_became_peered":"2024-03-16T12:19:44.551762+0000","last_unstale":"2024-03-16T12:20:12.776444+0000","last_undegraded":"2024-03-16T12:20:12.776444+0000","last_fullsized":"2024-03-16T12:20:12.776444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:24.454587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.727945+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:21:37.727945+0000","last_peered":"2024-03-16T12:21:37.727945+0000","last_clean":"2024-03-16T12:21:37.727945+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:21:37.727945+0000","last_undegraded":"2024-03-16T12:21:37.727945+0000","last_fullsized":"2024-03-16T12:21:37.727945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547991+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:19:44.547991+0000","last_peered":"2024-03-16T12:19:44.547991+0000","last_clean":"2024-03-16T12:19:44.547991+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:19:44.547991+0000","last_undegraded":"2024-03-16T12:19:44.547991+0000","last_fullsized":"2024-03-16T12:19:44.547991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790357+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:20:12.790357+0000","last_peered":"2024-03-16T12:20:12.790357+0000","last_clean":"2024-03-16T12:20:12.790357+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:20:12.790357+0000","last_undegraded":"2024-03-16T12:20:12.790357+0000","last_fullsized":"2024-03-16T12:20:12.790357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546518+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:44.546518+0000","last_peered":"2024-03-16T12:19:44.546518+0000","last_clean":"2024-03-16T12:19:44.546518+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:44.546518+0000","last_undegraded":"2024-03-16T12:19:44.546518+0000","last_fullsized":"2024-03-16T12:19:44.546518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548608+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:19:44.548608+0000","last_peered":"2024-03-16T12:19:44.548608+0000","last_clean":"2024-03-16T12:19:44.548608+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:19:44.548608+0000","last_undegraded":"2024-03-16T12:19:44.548608+0000","last_fullsized":"2024-03-16T12:19:44.548608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567444+0000","last_change":"2024-03-16T12:19:44.566987+0000","last_active":"2024-03-16T12:19:44.567444+0000","last_peered":"2024-03-16T12:19:44.567444+0000","last_clean":"2024-03-16T12:19:44.567444+0000","last_became_active":"2024-03-16T12:19:44.566529+0000","last_became_peered":"2024-03-16T12:19:44.566529+0000","last_unstale":"2024-03-16T12:19:44.567444+0000","last_undegraded":"2024-03-16T12:19:44.567444+0000","last_fullsized":"2024-03-16T12:19:44.567444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:53:03.124434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785697+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:20:12.785697+0000","last_peered":"2024-03-16T12:20:12.785697+0000","last_clean":"2024-03-16T12:20:12.785697+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:20:12.785697+0000","last_undegraded":"2024-03-16T12:20:12.785697+0000","last_fullsized":"2024-03-16T12:20:12.785697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548707+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:44.548707+0000","last_peered":"2024-03-16T12:19:44.548707+0000","last_clean":"2024-03-16T12:19:44.548707+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:44.548707+0000","last_undegraded":"2024-03-16T12:19:44.548707+0000","last_fullsized":"2024-03-16T12:19:44.548707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559891+0000","last_change":"2024-03-16T12:19:44.559544+0000","last_active":"2024-03-16T12:19:44.559891+0000","last_peered":"2024-03-16T12:19:44.559891+0000","last_clean":"2024-03-16T12:19:44.559891+0000","last_became_active":"2024-03-16T12:19:44.559030+0000","last_became_peered":"2024-03-16T12:19:44.559030+0000","last_unstale":"2024-03-16T12:19:44.559891+0000","last_undegraded":"2024-03-16T12:19:44.559891+0000","last_fullsized":"2024-03-16T12:19:44.559891+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:46.496512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7,6],"acting":[11,10,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543822+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:44.543822+0000","last_peered":"2024-03-16T12:19:44.543822+0000","last_clean":"2024-03-16T12:19:44.543822+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:44.543822+0000","last_undegraded":"2024-03-16T12:19:44.543822+0000","last_fullsized":"2024-03-16T12:19:44.543822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787665+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:20:12.787665+0000","last_peered":"2024-03-16T12:20:12.787665+0000","last_clean":"2024-03-16T12:20:12.787665+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:20:12.787665+0000","last_undegraded":"2024-03-16T12:20:12.787665+0000","last_fullsized":"2024-03-16T12:20:12.787665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546514+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:19:44.546514+0000","last_peered":"2024-03-16T12:19:44.546514+0000","last_clean":"2024-03-16T12:19:44.546514+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:19:44.546514+0000","last_undegraded":"2024-03-16T12:19:44.546514+0000","last_fullsized":"2024-03-16T12:19:44.546514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:50.091668+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:20:50.091668+0000","last_peered":"2024-03-16T12:20:50.091668+0000","last_clean":"2024-03-16T12:20:50.091668+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:20:50.091668+0000","last_undegraded":"2024-03-16T12:20:50.091668+0000","last_fullsized":"2024-03-16T12:20:50.091668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788697+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:20:12.788697+0000","last_peered":"2024-03-16T12:20:12.788697+0000","last_clean":"2024-03-16T12:20:12.788697+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:20:12.788697+0000","last_undegraded":"2024-03-16T12:20:12.788697+0000","last_fullsized":"2024-03-16T12:20:12.788697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546828+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:19:44.546828+0000","last_peered":"2024-03-16T12:19:44.546828+0000","last_clean":"2024-03-16T12:19:44.546828+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:19:44.546828+0000","last_undegraded":"2024-03-16T12:19:44.546828+0000","last_fullsized":"2024-03-16T12:19:44.546828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546979+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:44.546979+0000","last_peered":"2024-03-16T12:19:44.546979+0000","last_clean":"2024-03-16T12:19:44.546979+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:44.546979+0000","last_undegraded":"2024-03-16T12:19:44.546979+0000","last_fullsized":"2024-03-16T12:19:44.546979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804936+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:20:12.804936+0000","last_peered":"2024-03-16T12:20:12.804936+0000","last_clean":"2024-03-16T12:20:12.804936+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:20:12.804936+0000","last_undegraded":"2024-03-16T12:20:12.804936+0000","last_fullsized":"2024-03-16T12:20:12.804936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546486+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:19:44.546486+0000","last_peered":"2024-03-16T12:19:44.546486+0000","last_clean":"2024-03-16T12:19:44.546486+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:19:44.546486+0000","last_undegraded":"2024-03-16T12:19:44.546486+0000","last_fullsized":"2024-03-16T12:19:44.546486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168879+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:45.168879+0000","last_peered":"2024-03-16T12:19:45.168879+0000","last_clean":"2024-03-16T12:19:45.168879+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:45.168879+0000","last_undegraded":"2024-03-16T12:19:45.168879+0000","last_fullsized":"2024-03-16T12:19:45.168879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.748250+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:20:12.748250+0000","last_peered":"2024-03-16T12:20:12.748250+0000","last_clean":"2024-03-16T12:20:12.748250+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:20:12.748250+0000","last_undegraded":"2024-03-16T12:20:12.748250+0000","last_fullsized":"2024-03-16T12:20:12.748250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847167+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:19:44.847167+0000","last_peered":"2024-03-16T12:19:44.847167+0000","last_clean":"2024-03-16T12:19:44.847167+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:19:44.847167+0000","last_undegraded":"2024-03-16T12:19:44.847167+0000","last_fullsized":"2024-03-16T12:19:44.847167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546586+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:44.546586+0000","last_peered":"2024-03-16T12:19:44.546586+0000","last_clean":"2024-03-16T12:19:44.546586+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:44.546586+0000","last_undegraded":"2024-03-16T12:19:44.546586+0000","last_fullsized":"2024-03-16T12:19:44.546586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803505+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:20:12.803505+0000","last_peered":"2024-03-16T12:20:12.803505+0000","last_clean":"2024-03-16T12:20:12.803505+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:20:12.803505+0000","last_undegraded":"2024-03-16T12:20:12.803505+0000","last_fullsized":"2024-03-16T12:20:12.803505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546670+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:19:44.546670+0000","last_peered":"2024-03-16T12:19:44.546670+0000","last_clean":"2024-03-16T12:19:44.546670+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:19:44.546670+0000","last_undegraded":"2024-03-16T12:19:44.546670+0000","last_fullsized":"2024-03-16T12:19:44.546670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547065+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:19:44.547065+0000","last_peered":"2024-03-16T12:19:44.547065+0000","last_clean":"2024-03-16T12:19:44.547065+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:19:44.547065+0000","last_undegraded":"2024-03-16T12:19:44.547065+0000","last_fullsized":"2024-03-16T12:19:44.547065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805338+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:20:12.805338+0000","last_peered":"2024-03-16T12:20:12.805338+0000","last_clean":"2024-03-16T12:20:12.805338+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:20:12.805338+0000","last_undegraded":"2024-03-16T12:20:12.805338+0000","last_fullsized":"2024-03-16T12:20:12.805338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546343+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:19:44.546343+0000","last_peered":"2024-03-16T12:19:44.546343+0000","last_clean":"2024-03-16T12:19:44.546343+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:19:44.546343+0000","last_undegraded":"2024-03-16T12:19:44.546343+0000","last_fullsized":"2024-03-16T12:19:44.546343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555974+0000","last_change":"2024-03-16T12:19:44.555974+0000","last_active":"2024-03-16T12:19:44.555974+0000","last_peered":"2024-03-16T12:19:44.555974+0000","last_clean":"2024-03-16T12:19:44.555974+0000","last_became_active":"2024-03-16T12:19:44.550254+0000","last_became_peered":"2024-03-16T12:19:44.550254+0000","last_unstale":"2024-03-16T12:19:44.555974+0000","last_undegraded":"2024-03-16T12:19:44.555974+0000","last_fullsized":"2024-03-16T12:19:44.555974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789538+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:20:12.789538+0000","last_peered":"2024-03-16T12:20:12.789538+0000","last_clean":"2024-03-16T12:20:12.789538+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:20:12.789538+0000","last_undegraded":"2024-03-16T12:20:12.789538+0000","last_fullsized":"2024-03-16T12:20:12.789538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168880+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:19:45.168880+0000","last_peered":"2024-03-16T12:19:45.168880+0000","last_clean":"2024-03-16T12:19:45.168880+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:19:45.168880+0000","last_undegraded":"2024-03-16T12:19:45.168880+0000","last_fullsized":"2024-03-16T12:19:45.168880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548184+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:19:44.548184+0000","last_peered":"2024-03-16T12:19:44.548184+0000","last_clean":"2024-03-16T12:19:44.548184+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:19:44.548184+0000","last_undegraded":"2024-03-16T12:19:44.548184+0000","last_fullsized":"2024-03-16T12:19:44.548184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786719+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:20:12.786719+0000","last_peered":"2024-03-16T12:20:12.786719+0000","last_clean":"2024-03-16T12:20:12.786719+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:20:12.786719+0000","last_undegraded":"2024-03-16T12:20:12.786719+0000","last_fullsized":"2024-03-16T12:20:12.786719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172049+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:19:45.172049+0000","last_peered":"2024-03-16T12:19:45.172049+0000","last_clean":"2024-03-16T12:19:45.172049+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:19:45.172049+0000","last_undegraded":"2024-03-16T12:19:45.172049+0000","last_fullsized":"2024-03-16T12:19:45.172049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550000+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:19:44.550000+0000","last_peered":"2024-03-16T12:19:44.550000+0000","last_clean":"2024-03-16T12:19:44.550000+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:19:44.550000+0000","last_undegraded":"2024-03-16T12:19:44.550000+0000","last_fullsized":"2024-03-16T12:19:44.550000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791712+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:20:12.791712+0000","last_peered":"2024-03-16T12:20:12.791712+0000","last_clean":"2024-03-16T12:20:12.791712+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:20:12.791712+0000","last_undegraded":"2024-03-16T12:20:12.791712+0000","last_fullsized":"2024-03-16T12:20:12.791712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549274+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:19:44.549274+0000","last_peered":"2024-03-16T12:19:44.549274+0000","last_clean":"2024-03-16T12:19:44.549274+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:19:44.549274+0000","last_undegraded":"2024-03-16T12:19:44.549274+0000","last_fullsized":"2024-03-16T12:19:44.549274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559640+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:19:44.559640+0000","last_peered":"2024-03-16T12:19:44.559640+0000","last_clean":"2024-03-16T12:19:44.559640+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:19:44.559640+0000","last_undegraded":"2024-03-16T12:19:44.559640+0000","last_fullsized":"2024-03-16T12:19:44.559640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.771744+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:20:12.771744+0000","last_peered":"2024-03-16T12:20:12.771744+0000","last_clean":"2024-03-16T12:20:12.771744+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:20:12.771744+0000","last_undegraded":"2024-03-16T12:20:12.771744+0000","last_fullsized":"2024-03-16T12:20:12.771744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774255+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:20:12.774255+0000","last_peered":"2024-03-16T12:20:12.774255+0000","last_clean":"2024-03-16T12:20:12.774255+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:20:12.774255+0000","last_undegraded":"2024-03-16T12:20:12.774255+0000","last_fullsized":"2024-03-16T12:20:12.774255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781112+0000","last_change":"2024-03-16T12:19:44.847899+0000","last_active":"2024-03-16T12:20:12.781112+0000","last_peered":"2024-03-16T12:20:12.781112+0000","last_clean":"2024-03-16T12:20:12.781112+0000","last_became_active":"2024-03-16T12:19:44.847589+0000","last_became_peered":"2024-03-16T12:19:44.847589+0000","last_unstale":"2024-03-16T12:20:12.781112+0000","last_undegraded":"2024-03-16T12:20:12.781112+0000","last_fullsized":"2024-03-16T12:20:12.781112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:59.091516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.802828+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:20:12.802828+0000","last_peered":"2024-03-16T12:20:12.802828+0000","last_clean":"2024-03-16T12:20:12.802828+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:20:12.802828+0000","last_undegraded":"2024-03-16T12:20:12.802828+0000","last_fullsized":"2024-03-16T12:20:12.802828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804518+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:20:12.804518+0000","last_peered":"2024-03-16T12:20:12.804518+0000","last_clean":"2024-03-16T12:20:12.804518+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:20:12.804518+0000","last_undegraded":"2024-03-16T12:20:12.804518+0000","last_fullsized":"2024-03-16T12:20:12.804518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805439+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:20:12.805439+0000","last_peered":"2024-03-16T12:20:12.805439+0000","last_clean":"2024-03-16T12:20:12.805439+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:20:12.805439+0000","last_undegraded":"2024-03-16T12:20:12.805439+0000","last_fullsized":"2024-03-16T12:20:12.805439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770556+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:20:12.770556+0000","last_peered":"2024-03-16T12:20:12.770556+0000","last_clean":"2024-03-16T12:20:12.770556+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:20:12.770556+0000","last_undegraded":"2024-03-16T12:20:12.770556+0000","last_fullsized":"2024-03-16T12:20:12.770556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776877+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:20:12.776877+0000","last_peered":"2024-03-16T12:20:12.776877+0000","last_clean":"2024-03-16T12:20:12.776877+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:20:12.776877+0000","last_undegraded":"2024-03-16T12:20:12.776877+0000","last_fullsized":"2024-03-16T12:20:12.776877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.777318+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:20:12.777318+0000","last_peered":"2024-03-16T12:20:12.777318+0000","last_clean":"2024-03-16T12:20:12.777318+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:20:12.777318+0000","last_undegraded":"2024-03-16T12:20:12.777318+0000","last_fullsized":"2024-03-16T12:20:12.777318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799957+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:20:12.799957+0000","last_peered":"2024-03-16T12:20:12.799957+0000","last_clean":"2024-03-16T12:20:12.799957+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:20:12.799957+0000","last_undegraded":"2024-03-16T12:20:12.799957+0000","last_fullsized":"2024-03-16T12:20:12.799957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.760742+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:20:12.760742+0000","last_peered":"2024-03-16T12:20:12.760742+0000","last_clean":"2024-03-16T12:20:12.760742+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:20:12.760742+0000","last_undegraded":"2024-03-16T12:20:12.760742+0000","last_fullsized":"2024-03-16T12:20:12.760742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800264+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:20:12.800264+0000","last_peered":"2024-03-16T12:20:12.800264+0000","last_clean":"2024-03-16T12:20:12.800264+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:20:12.800264+0000","last_undegraded":"2024-03-16T12:20:12.800264+0000","last_fullsized":"2024-03-16T12:20:12.800264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.806609+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:20:12.806609+0000","last_peered":"2024-03-16T12:20:12.806609+0000","last_clean":"2024-03-16T12:20:12.806609+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:20:12.806609+0000","last_undegraded":"2024-03-16T12:20:12.806609+0000","last_fullsized":"2024-03-16T12:20:12.806609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781562+0000","last_change":"2024-03-16T12:19:44.849288+0000","last_active":"2024-03-16T12:20:12.781562+0000","last_peered":"2024-03-16T12:20:12.781562+0000","last_clean":"2024-03-16T12:20:12.781562+0000","last_became_active":"2024-03-16T12:19:44.848869+0000","last_became_peered":"2024-03-16T12:19:44.848869+0000","last_unstale":"2024-03-16T12:20:12.781562+0000","last_undegraded":"2024-03-16T12:20:12.781562+0000","last_fullsized":"2024-03-16T12:20:12.781562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:36.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72185,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":929,"num_read_kb":541,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":216555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":500849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28696,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716456,"statfs":{"total":95995035648,"available":95965650944,"internally_reserved":0,"allocated":1720320,"data_stored":1156658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":500971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1096799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495693,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":564402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":565718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":501187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1092077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:21:39.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T12:21:39.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:38 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3668691285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:39.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:38 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/822617418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 -- 172.21.15.98:0/2021426699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c103a00 msgr2=0x7f2a9c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 --2- 172.21.15.98:0/2021426699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c103a00 0x7f2a9c103e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009900 tx=0x7f2a8c02bc40 comp rx=0 tx=0).stop 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 -- 172.21.15.98:0/2021426699 shutdown_connections 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 --2- 172.21.15.98:0/2021426699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 --2- 172.21.15.98:0/2021426699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 --2- 172.21.15.98:0/2021426699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c103a00 0x7f2a9c103e20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 -- 172.21.15.98:0/2021426699 >> 172.21.15.98:0/2021426699 conn(0x7f2a9c0fef90 msgr2=0x7f2a9c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.594+0000 7f2aa1400640 1 -- 172.21.15.98:0/2021426699 shutdown_connections 2024-03-16T12:21:39.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.595+0000 7f2aa1400640 1 -- 172.21.15.98:0/2021426699 wait complete. 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.595+0000 7f2aa1400640 1 Processor -- start 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 -- start start 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 0x7f2a9c199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10f6a0 con 0x7f2a9c103a00 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10f520 con 0x7f2a9c1055b0 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2aa1400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10f820 con 0x7f2a9c104bf0 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 0x7f2a9c199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a99e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a9b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a99e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56202/0 (socket says 172.21.15.98:56202) 2024-03-16T12:21:39.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a99e00640 1 -- 172.21.15.98:0/3988859862 learned_addr learned my addr 172.21.15.98:0/3988859862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.596+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 0x7f2a9c199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43104/0 (socket says 172.21.15.98:43104) 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 -- 172.21.15.98:0/3988859862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 msgr2=0x7f2a9c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 -- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 msgr2=0x7f2a9c199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 0x7f2a9c199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 -- 172.21.15.98:0/3988859862 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c1a1900 con 0x7f2a9c1055b0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a99e00640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a9b200640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c1a1260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ec50 tx=0x7f2a9400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94003b70 con 0x7f2a9c1055b0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a94003d10 con 0x7f2a9c1055b0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9c1a1bf0 con 0x7f2a9c1055b0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94010890 con 0x7f2a9c1055b0 2024-03-16T12:21:39.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.597+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9c1a2110 con 0x7f2a9c1055b0 2024-03-16T12:21:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.598+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9c10ad20 con 0x7f2a9c1055b0 2024-03-16T12:21:39.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.600+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2a94003fc0 con 0x7f2a9c1055b0 2024-03-16T12:21:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.601+0000 7f2a93400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 0x7f2a800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.602+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2a940cf320 con 0x7f2a9c1055b0 2024-03-16T12:21:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.602+0000 7f2a9a800640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 0x7f2a800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:39.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.602+0000 7f2a9a800640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 0x7f2a800a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c02c180 tx=0x7f2a8c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:39.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.605+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a9409c000 con 0x7f2a9c1055b0 2024-03-16T12:21:39.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.803+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a9c061c20 con 0x7f2a800a2880 2024-03-16T12:21:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.831+0000 7f2a93400640 1 -- 172.21.15.98:0/3988859862 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340494 (secure 0 0 0) 0x7f2a9c061c20 con 0x7f2a800a2880 2024-03-16T12:21:39.815 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:39.820 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.839+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 msgr2=0x7f2a800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.839+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 0x7f2a800a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c02c180 tx=0x7f2a8c02cf90 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.839+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 msgr2=0x7f2a9c1a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.839+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c1a1260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ec50 tx=0x7f2a9400c6a0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 shutdown_connections 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2a800a2880 0x7f2a800a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a9c1055b0 0x7f2a9c1a1260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a9c104bf0 0x7f2a9c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 --2- 172.21.15.98:0/3988859862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c103a00 0x7f2a9c199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 >> 172.21.15.98:0/3988859862 conn(0x7f2a9c0fef90 msgr2=0x7f2a9c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 shutdown_connections 2024-03-16T12:21:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:39.840+0000 7f2aa1400640 1 -- 172.21.15.98:0/3988859862 wait complete. 2024-03-16T12:21:39.843 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-16T12:21:38.851289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662787,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1055,"num_read_kb":650,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384060,"kb_used_data":14108,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557764,"statfs":{"total":1151940427776,"available":1151547150336,"internally_reserved":0,"allocated":14446592,"data_stored":7974071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5993,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":43,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004627"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803803+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:20:12.803803+0000","last_peered":"2024-03-16T12:20:12.803803+0000","last_clean":"2024-03-16T12:20:12.803803+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:20:12.803803+0000","last_undegraded":"2024-03-16T12:20:12.803803+0000","last_fullsized":"2024-03-16T12:20:12.803803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798435+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:20:12.798435+0000","last_peered":"2024-03-16T12:20:12.798435+0000","last_clean":"2024-03-16T12:20:12.798435+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:20:12.798435+0000","last_undegraded":"2024-03-16T12:20:12.798435+0000","last_fullsized":"2024-03-16T12:20:12.798435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800413+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:20:12.800413+0000","last_peered":"2024-03-16T12:20:12.800413+0000","last_clean":"2024-03-16T12:20:12.800413+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:20:12.800413+0000","last_undegraded":"2024-03-16T12:20:12.800413+0000","last_fullsized":"2024-03-16T12:20:12.800413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755748+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:20:12.755748+0000","last_peered":"2024-03-16T12:20:12.755748+0000","last_clean":"2024-03-16T12:20:12.755748+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:20:12.755748+0000","last_undegraded":"2024-03-16T12:20:12.755748+0000","last_fullsized":"2024-03-16T12:20:12.755748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791944+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:20:12.791944+0000","last_peered":"2024-03-16T12:20:12.791944+0000","last_clean":"2024-03-16T12:20:12.791944+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:20:12.791944+0000","last_undegraded":"2024-03-16T12:20:12.791944+0000","last_fullsized":"2024-03-16T12:20:12.791944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785928+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:20:12.785928+0000","last_peered":"2024-03-16T12:20:12.785928+0000","last_clean":"2024-03-16T12:20:12.785928+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:20:12.785928+0000","last_undegraded":"2024-03-16T12:20:12.785928+0000","last_fullsized":"2024-03-16T12:20:12.785928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.773384+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:20:12.773384+0000","last_peered":"2024-03-16T12:20:12.773384+0000","last_clean":"2024-03-16T12:20:12.773384+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:20:12.773384+0000","last_undegraded":"2024-03-16T12:20:12.773384+0000","last_fullsized":"2024-03-16T12:20:12.773384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793133+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:20:12.793133+0000","last_peered":"2024-03-16T12:20:12.793133+0000","last_clean":"2024-03-16T12:20:12.793133+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:20:12.793133+0000","last_undegraded":"2024-03-16T12:20:12.793133+0000","last_fullsized":"2024-03-16T12:20:12.793133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785095+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:20:12.785095+0000","last_peered":"2024-03-16T12:20:12.785095+0000","last_clean":"2024-03-16T12:20:12.785095+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:20:12.785095+0000","last_undegraded":"2024-03-16T12:20:12.785095+0000","last_fullsized":"2024-03-16T12:20:12.785095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785364+0000","last_change":"2024-03-16T12:19:44.848697+0000","last_active":"2024-03-16T12:20:12.785364+0000","last_peered":"2024-03-16T12:20:12.785364+0000","last_clean":"2024-03-16T12:20:12.785364+0000","last_became_active":"2024-03-16T12:19:44.848414+0000","last_became_peered":"2024-03-16T12:19:44.848414+0000","last_unstale":"2024-03-16T12:20:12.785364+0000","last_undegraded":"2024-03-16T12:20:12.785364+0000","last_fullsized":"2024-03-16T12:20:12.785364+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:54.721056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786403+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:20:12.786403+0000","last_peered":"2024-03-16T12:20:12.786403+0000","last_clean":"2024-03-16T12:20:12.786403+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:20:12.786403+0000","last_undegraded":"2024-03-16T12:20:12.786403+0000","last_fullsized":"2024-03-16T12:20:12.786403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755018+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:20:12.755018+0000","last_peered":"2024-03-16T12:20:12.755018+0000","last_clean":"2024-03-16T12:20:12.755018+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:20:12.755018+0000","last_undegraded":"2024-03-16T12:20:12.755018+0000","last_fullsized":"2024-03-16T12:20:12.755018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790423+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:20:12.790423+0000","last_peered":"2024-03-16T12:20:12.790423+0000","last_clean":"2024-03-16T12:20:12.790423+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:20:12.790423+0000","last_undegraded":"2024-03-16T12:20:12.790423+0000","last_fullsized":"2024-03-16T12:20:12.790423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794049+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:20:12.794049+0000","last_peered":"2024-03-16T12:20:12.794049+0000","last_clean":"2024-03-16T12:20:12.794049+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:20:12.794049+0000","last_undegraded":"2024-03-16T12:20:12.794049+0000","last_fullsized":"2024-03-16T12:20:12.794049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.795950+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:20:12.795950+0000","last_peered":"2024-03-16T12:20:12.795950+0000","last_clean":"2024-03-16T12:20:12.795950+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:20:12.795950+0000","last_undegraded":"2024-03-16T12:20:12.795950+0000","last_fullsized":"2024-03-16T12:20:12.795950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803918+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:20:12.803918+0000","last_peered":"2024-03-16T12:20:12.803918+0000","last_clean":"2024-03-16T12:20:12.803918+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:20:12.803918+0000","last_undegraded":"2024-03-16T12:20:12.803918+0000","last_fullsized":"2024-03-16T12:20:12.803918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804028+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:20:12.804028+0000","last_peered":"2024-03-16T12:20:12.804028+0000","last_clean":"2024-03-16T12:20:12.804028+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:20:12.804028+0000","last_undegraded":"2024-03-16T12:20:12.804028+0000","last_fullsized":"2024-03-16T12:20:12.804028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786205+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:20:12.786205+0000","last_peered":"2024-03-16T12:20:12.786205+0000","last_clean":"2024-03-16T12:20:12.786205+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:20:12.786205+0000","last_undegraded":"2024-03-16T12:20:12.786205+0000","last_fullsized":"2024-03-16T12:20:12.786205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792899+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:20:12.792899+0000","last_peered":"2024-03-16T12:20:12.792899+0000","last_clean":"2024-03-16T12:20:12.792899+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:20:12.792899+0000","last_undegraded":"2024-03-16T12:20:12.792899+0000","last_fullsized":"2024-03-16T12:20:12.792899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547296+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:19:44.547296+0000","last_peered":"2024-03-16T12:19:44.547296+0000","last_clean":"2024-03-16T12:19:44.547296+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:19:44.547296+0000","last_undegraded":"2024-03-16T12:19:44.547296+0000","last_fullsized":"2024-03-16T12:19:44.547296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847301+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:44.847301+0000","last_peered":"2024-03-16T12:19:44.847301+0000","last_clean":"2024-03-16T12:19:44.847301+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:44.847301+0000","last_undegraded":"2024-03-16T12:19:44.847301+0000","last_fullsized":"2024-03-16T12:19:44.847301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.769143+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:20:12.769143+0000","last_peered":"2024-03-16T12:20:12.769143+0000","last_clean":"2024-03-16T12:20:12.769143+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:20:12.769143+0000","last_undegraded":"2024-03-16T12:20:12.769143+0000","last_fullsized":"2024-03-16T12:20:12.769143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559668+0000","last_change":"2024-03-16T12:19:44.559192+0000","last_active":"2024-03-16T12:19:44.559668+0000","last_peered":"2024-03-16T12:19:44.559668+0000","last_clean":"2024-03-16T12:19:44.559668+0000","last_became_active":"2024-03-16T12:19:44.558795+0000","last_became_peered":"2024-03-16T12:19:44.558795+0000","last_unstale":"2024-03-16T12:19:44.559668+0000","last_undegraded":"2024-03-16T12:19:44.559668+0000","last_fullsized":"2024-03-16T12:19:44.559668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:41:28.267737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.552034+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:19:44.552034+0000","last_peered":"2024-03-16T12:19:44.552034+0000","last_clean":"2024-03-16T12:19:44.552034+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:19:44.552034+0000","last_undegraded":"2024-03-16T12:19:44.552034+0000","last_fullsized":"2024-03-16T12:19:44.552034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804064+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:20:12.804064+0000","last_peered":"2024-03-16T12:20:12.804064+0000","last_clean":"2024-03-16T12:20:12.804064+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:20:12.804064+0000","last_undegraded":"2024-03-16T12:20:12.804064+0000","last_fullsized":"2024-03-16T12:20:12.804064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546380+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:19:44.546380+0000","last_peered":"2024-03-16T12:19:44.546380+0000","last_clean":"2024-03-16T12:19:44.546380+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:19:44.546380+0000","last_undegraded":"2024-03-16T12:19:44.546380+0000","last_fullsized":"2024-03-16T12:19:44.546380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543851+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:44.543851+0000","last_peered":"2024-03-16T12:19:44.543851+0000","last_clean":"2024-03-16T12:19:44.543851+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:44.543851+0000","last_undegraded":"2024-03-16T12:19:44.543851+0000","last_fullsized":"2024-03-16T12:19:44.543851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789084+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:20:12.789084+0000","last_peered":"2024-03-16T12:20:12.789084+0000","last_clean":"2024-03-16T12:20:12.789084+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:20:12.789084+0000","last_undegraded":"2024-03-16T12:20:12.789084+0000","last_fullsized":"2024-03-16T12:20:12.789084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546007+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:19:44.546007+0000","last_peered":"2024-03-16T12:19:44.546007+0000","last_clean":"2024-03-16T12:19:44.546007+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:19:44.546007+0000","last_undegraded":"2024-03-16T12:19:44.546007+0000","last_fullsized":"2024-03-16T12:19:44.546007+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543938+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:19:44.543938+0000","last_peered":"2024-03-16T12:19:44.543938+0000","last_clean":"2024-03-16T12:19:44.543938+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:19:44.543938+0000","last_undegraded":"2024-03-16T12:19:44.543938+0000","last_fullsized":"2024-03-16T12:19:44.543938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.784812+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:20:12.784812+0000","last_peered":"2024-03-16T12:20:12.784812+0000","last_clean":"2024-03-16T12:20:12.784812+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:20:12.784812+0000","last_undegraded":"2024-03-16T12:20:12.784812+0000","last_fullsized":"2024-03-16T12:20:12.784812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547794+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:19:44.547794+0000","last_peered":"2024-03-16T12:19:44.547794+0000","last_clean":"2024-03-16T12:19:44.547794+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:19:44.547794+0000","last_undegraded":"2024-03-16T12:19:44.547794+0000","last_fullsized":"2024-03-16T12:19:44.547794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":187,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.644775+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:21:37.644775+0000","last_peered":"2024-03-16T12:21:37.644775+0000","last_clean":"2024-03-16T12:21:37.644775+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:21:37.644775+0000","last_undegraded":"2024-03-16T12:21:37.644775+0000","last_fullsized":"2024-03-16T12:21:37.644775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781031+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:20:12.781031+0000","last_peered":"2024-03-16T12:20:12.781031+0000","last_clean":"2024-03-16T12:20:12.781031+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:20:12.781031+0000","last_undegraded":"2024-03-16T12:20:12.781031+0000","last_fullsized":"2024-03-16T12:20:12.781031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548011+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:19:44.548011+0000","last_peered":"2024-03-16T12:19:44.548011+0000","last_clean":"2024-03-16T12:19:44.548011+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:19:44.548011+0000","last_undegraded":"2024-03-16T12:19:44.548011+0000","last_fullsized":"2024-03-16T12:19:44.548011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547430+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:19:44.547430+0000","last_peered":"2024-03-16T12:19:44.547430+0000","last_clean":"2024-03-16T12:19:44.547430+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:19:44.547430+0000","last_undegraded":"2024-03-16T12:19:44.547430+0000","last_fullsized":"2024-03-16T12:19:44.547430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798746+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:20:12.798746+0000","last_peered":"2024-03-16T12:20:12.798746+0000","last_clean":"2024-03-16T12:20:12.798746+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:20:12.798746+0000","last_undegraded":"2024-03-16T12:20:12.798746+0000","last_fullsized":"2024-03-16T12:20:12.798746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545612+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:19:44.545612+0000","last_peered":"2024-03-16T12:19:44.545612+0000","last_clean":"2024-03-16T12:19:44.545612+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:19:44.545612+0000","last_undegraded":"2024-03-16T12:19:44.545612+0000","last_fullsized":"2024-03-16T12:19:44.545612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555984+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:44.555984+0000","last_peered":"2024-03-16T12:19:44.555984+0000","last_clean":"2024-03-16T12:19:44.555984+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:44.555984+0000","last_undegraded":"2024-03-16T12:19:44.555984+0000","last_fullsized":"2024-03-16T12:19:44.555984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792926+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:20:12.792926+0000","last_peered":"2024-03-16T12:20:12.792926+0000","last_clean":"2024-03-16T12:20:12.792926+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:20:12.792926+0000","last_undegraded":"2024-03-16T12:20:12.792926+0000","last_fullsized":"2024-03-16T12:20:12.792926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172396+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:19:45.172396+0000","last_peered":"2024-03-16T12:19:45.172396+0000","last_clean":"2024-03-16T12:19:45.172396+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:19:45.172396+0000","last_undegraded":"2024-03-16T12:19:45.172396+0000","last_fullsized":"2024-03-16T12:19:45.172396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549572+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:19:44.549572+0000","last_peered":"2024-03-16T12:19:44.549572+0000","last_clean":"2024-03-16T12:19:44.549572+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:19:44.549572+0000","last_undegraded":"2024-03-16T12:19:44.549572+0000","last_fullsized":"2024-03-16T12:19:44.549572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.801975+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:20:12.801975+0000","last_peered":"2024-03-16T12:20:12.801975+0000","last_clean":"2024-03-16T12:20:12.801975+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:20:12.801975+0000","last_undegraded":"2024-03-16T12:20:12.801975+0000","last_fullsized":"2024-03-16T12:20:12.801975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546523+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:19:44.546523+0000","last_peered":"2024-03-16T12:19:44.546523+0000","last_clean":"2024-03-16T12:19:44.546523+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:19:44.546523+0000","last_undegraded":"2024-03-16T12:19:44.546523+0000","last_fullsized":"2024-03-16T12:19:44.546523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":181,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:35.641987+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:21:35.641987+0000","last_peered":"2024-03-16T12:21:35.641987+0000","last_clean":"2024-03-16T12:21:35.641987+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:21:35.641987+0000","last_undegraded":"2024-03-16T12:21:35.641987+0000","last_fullsized":"2024-03-16T12:21:35.641987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787423+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:20:12.787423+0000","last_peered":"2024-03-16T12:20:12.787423+0000","last_clean":"2024-03-16T12:20:12.787423+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:20:12.787423+0000","last_undegraded":"2024-03-16T12:20:12.787423+0000","last_fullsized":"2024-03-16T12:20:12.787423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845673+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:19:44.845673+0000","last_peered":"2024-03-16T12:19:44.845673+0000","last_clean":"2024-03-16T12:19:44.845673+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:19:44.845673+0000","last_undegraded":"2024-03-16T12:19:44.845673+0000","last_fullsized":"2024-03-16T12:19:44.845673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548131+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:44.548131+0000","last_peered":"2024-03-16T12:19:44.548131+0000","last_clean":"2024-03-16T12:19:44.548131+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:44.548131+0000","last_undegraded":"2024-03-16T12:19:44.548131+0000","last_fullsized":"2024-03-16T12:19:44.548131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794150+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:20:12.794150+0000","last_peered":"2024-03-16T12:20:12.794150+0000","last_clean":"2024-03-16T12:20:12.794150+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:20:12.794150+0000","last_undegraded":"2024-03-16T12:20:12.794150+0000","last_fullsized":"2024-03-16T12:20:12.794150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546597+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:19:44.546597+0000","last_peered":"2024-03-16T12:19:44.546597+0000","last_clean":"2024-03-16T12:19:44.546597+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:19:44.546597+0000","last_undegraded":"2024-03-16T12:19:44.546597+0000","last_fullsized":"2024-03-16T12:19:44.546597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549488+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:44.549488+0000","last_peered":"2024-03-16T12:19:44.549488+0000","last_clean":"2024-03-16T12:19:44.549488+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:44.549488+0000","last_undegraded":"2024-03-16T12:19:44.549488+0000","last_fullsized":"2024-03-16T12:19:44.549488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785886+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:20:12.785886+0000","last_peered":"2024-03-16T12:20:12.785886+0000","last_clean":"2024-03-16T12:20:12.785886+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:20:12.785886+0000","last_undegraded":"2024-03-16T12:20:12.785886+0000","last_fullsized":"2024-03-16T12:20:12.785886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845685+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:19:44.845685+0000","last_peered":"2024-03-16T12:19:44.845685+0000","last_clean":"2024-03-16T12:19:44.845685+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:19:44.845685+0000","last_undegraded":"2024-03-16T12:19:44.845685+0000","last_fullsized":"2024-03-16T12:19:44.845685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'40","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:36.643509+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:21:36.643509+0000","last_peered":"2024-03-16T12:21:36.643509+0000","last_clean":"2024-03-16T12:21:36.643509+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:21:36.643509+0000","last_undegraded":"2024-03-16T12:21:36.643509+0000","last_fullsized":"2024-03-16T12:21:36.643509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68265,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":222,"num_write":41,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.753180+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:20:12.753180+0000","last_peered":"2024-03-16T12:20:12.753180+0000","last_clean":"2024-03-16T12:20:12.753180+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:20:12.753180+0000","last_undegraded":"2024-03-16T12:20:12.753180+0000","last_fullsized":"2024-03-16T12:20:12.753180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550225+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:19:44.550225+0000","last_peered":"2024-03-16T12:19:44.550225+0000","last_clean":"2024-03-16T12:19:44.550225+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:19:44.550225+0000","last_undegraded":"2024-03-16T12:19:44.550225+0000","last_fullsized":"2024-03-16T12:19:44.550225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565318+0000","last_change":"2024-03-16T12:19:44.565318+0000","last_active":"2024-03-16T12:19:44.565318+0000","last_peered":"2024-03-16T12:19:44.565318+0000","last_clean":"2024-03-16T12:19:44.565318+0000","last_became_active":"2024-03-16T12:19:44.551024+0000","last_became_peered":"2024-03-16T12:19:44.551024+0000","last_unstale":"2024-03-16T12:19:44.565318+0000","last_undegraded":"2024-03-16T12:19:44.565318+0000","last_fullsized":"2024-03-16T12:19:44.565318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:15.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800693+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:20:12.800693+0000","last_peered":"2024-03-16T12:20:12.800693+0000","last_clean":"2024-03-16T12:20:12.800693+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:20:12.800693+0000","last_undegraded":"2024-03-16T12:20:12.800693+0000","last_fullsized":"2024-03-16T12:20:12.800693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545017+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:19:44.545017+0000","last_peered":"2024-03-16T12:19:44.545017+0000","last_clean":"2024-03-16T12:19:44.545017+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:19:44.545017+0000","last_undegraded":"2024-03-16T12:19:44.545017+0000","last_fullsized":"2024-03-16T12:19:44.545017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549187+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:44.549187+0000","last_peered":"2024-03-16T12:19:44.549187+0000","last_clean":"2024-03-16T12:19:44.549187+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:44.549187+0000","last_undegraded":"2024-03-16T12:19:44.549187+0000","last_fullsized":"2024-03-16T12:19:44.549187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790351+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:20:12.790351+0000","last_peered":"2024-03-16T12:20:12.790351+0000","last_clean":"2024-03-16T12:20:12.790351+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:20:12.790351+0000","last_undegraded":"2024-03-16T12:20:12.790351+0000","last_fullsized":"2024-03-16T12:20:12.790351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.846970+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:19:44.846970+0000","last_peered":"2024-03-16T12:19:44.846970+0000","last_clean":"2024-03-16T12:19:44.846970+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:19:44.846970+0000","last_undegraded":"2024-03-16T12:19:44.846970+0000","last_fullsized":"2024-03-16T12:19:44.846970+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":182,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:32.719508+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:21:32.719508+0000","last_peered":"2024-03-16T12:21:32.719508+0000","last_clean":"2024-03-16T12:21:32.719508+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:21:32.719508+0000","last_undegraded":"2024-03-16T12:21:32.719508+0000","last_fullsized":"2024-03-16T12:21:32.719508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":898,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755629+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:20:12.755629+0000","last_peered":"2024-03-16T12:20:12.755629+0000","last_clean":"2024-03-16T12:20:12.755629+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:20:12.755629+0000","last_undegraded":"2024-03-16T12:20:12.755629+0000","last_fullsized":"2024-03-16T12:20:12.755629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567169+0000","last_change":"2024-03-16T12:19:44.565998+0000","last_active":"2024-03-16T12:19:44.567169+0000","last_peered":"2024-03-16T12:19:44.567169+0000","last_clean":"2024-03-16T12:19:44.567169+0000","last_became_active":"2024-03-16T12:19:44.564591+0000","last_became_peered":"2024-03-16T12:19:44.564591+0000","last_unstale":"2024-03-16T12:19:44.567169+0000","last_undegraded":"2024-03-16T12:19:44.567169+0000","last_fullsized":"2024-03-16T12:19:44.567169+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:57:48.227439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543368+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:44.543368+0000","last_peered":"2024-03-16T12:19:44.543368+0000","last_clean":"2024-03-16T12:19:44.543368+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:44.543368+0000","last_undegraded":"2024-03-16T12:19:44.543368+0000","last_fullsized":"2024-03-16T12:19:44.543368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782622+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:20:12.782622+0000","last_peered":"2024-03-16T12:20:12.782622+0000","last_clean":"2024-03-16T12:20:12.782622+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:20:12.782622+0000","last_undegraded":"2024-03-16T12:20:12.782622+0000","last_fullsized":"2024-03-16T12:20:12.782622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548932+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:19:44.548932+0000","last_peered":"2024-03-16T12:19:44.548932+0000","last_clean":"2024-03-16T12:19:44.548932+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:19:44.548932+0000","last_undegraded":"2024-03-16T12:19:44.548932+0000","last_fullsized":"2024-03-16T12:19:44.548932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543880+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:44.543880+0000","last_peered":"2024-03-16T12:19:44.543880+0000","last_clean":"2024-03-16T12:19:44.543880+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:44.543880+0000","last_undegraded":"2024-03-16T12:19:44.543880+0000","last_fullsized":"2024-03-16T12:19:44.543880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770720+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:20:12.770720+0000","last_peered":"2024-03-16T12:20:12.770720+0000","last_clean":"2024-03-16T12:20:12.770720+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:20:12.770720+0000","last_undegraded":"2024-03-16T12:20:12.770720+0000","last_fullsized":"2024-03-16T12:20:12.770720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543490+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:19:44.543490+0000","last_peered":"2024-03-16T12:19:44.543490+0000","last_clean":"2024-03-16T12:19:44.543490+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:19:44.543490+0000","last_undegraded":"2024-03-16T12:19:44.543490+0000","last_fullsized":"2024-03-16T12:19:44.543490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847452+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:19:44.847452+0000","last_peered":"2024-03-16T12:19:44.847452+0000","last_clean":"2024-03-16T12:19:44.847452+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:19:44.847452+0000","last_undegraded":"2024-03-16T12:19:44.847452+0000","last_fullsized":"2024-03-16T12:19:44.847452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793183+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:20:12.793183+0000","last_peered":"2024-03-16T12:20:12.793183+0000","last_clean":"2024-03-16T12:20:12.793183+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:20:12.793183+0000","last_undegraded":"2024-03-16T12:20:12.793183+0000","last_fullsized":"2024-03-16T12:20:12.793183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559770+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:19:44.559770+0000","last_peered":"2024-03-16T12:19:44.559770+0000","last_clean":"2024-03-16T12:19:44.559770+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:19:44.559770+0000","last_undegraded":"2024-03-16T12:19:44.559770+0000","last_fullsized":"2024-03-16T12:19:44.559770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547822+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:19:44.547822+0000","last_peered":"2024-03-16T12:19:44.547822+0000","last_clean":"2024-03-16T12:19:44.547822+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:19:44.547822+0000","last_undegraded":"2024-03-16T12:19:44.547822+0000","last_fullsized":"2024-03-16T12:19:44.547822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786284+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:20:12.786284+0000","last_peered":"2024-03-16T12:20:12.786284+0000","last_clean":"2024-03-16T12:20:12.786284+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:20:12.786284+0000","last_undegraded":"2024-03-16T12:20:12.786284+0000","last_fullsized":"2024-03-16T12:20:12.786284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559484+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:19:44.559484+0000","last_peered":"2024-03-16T12:19:44.559484+0000","last_clean":"2024-03-16T12:19:44.559484+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:19:44.559484+0000","last_undegraded":"2024-03-16T12:19:44.559484+0000","last_fullsized":"2024-03-16T12:19:44.559484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776444+0000","last_change":"2024-03-16T12:19:44.552241+0000","last_active":"2024-03-16T12:20:12.776444+0000","last_peered":"2024-03-16T12:20:12.776444+0000","last_clean":"2024-03-16T12:20:12.776444+0000","last_became_active":"2024-03-16T12:19:44.551762+0000","last_became_peered":"2024-03-16T12:19:44.551762+0000","last_unstale":"2024-03-16T12:20:12.776444+0000","last_undegraded":"2024-03-16T12:20:12.776444+0000","last_fullsized":"2024-03-16T12:20:12.776444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:24.454587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.727945+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:21:37.727945+0000","last_peered":"2024-03-16T12:21:37.727945+0000","last_clean":"2024-03-16T12:21:37.727945+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:21:37.727945+0000","last_undegraded":"2024-03-16T12:21:37.727945+0000","last_fullsized":"2024-03-16T12:21:37.727945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547991+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:19:44.547991+0000","last_peered":"2024-03-16T12:19:44.547991+0000","last_clean":"2024-03-16T12:19:44.547991+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:19:44.547991+0000","last_undegraded":"2024-03-16T12:19:44.547991+0000","last_fullsized":"2024-03-16T12:19:44.547991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790357+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:20:12.790357+0000","last_peered":"2024-03-16T12:20:12.790357+0000","last_clean":"2024-03-16T12:20:12.790357+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:20:12.790357+0000","last_undegraded":"2024-03-16T12:20:12.790357+0000","last_fullsized":"2024-03-16T12:20:12.790357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546518+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:44.546518+0000","last_peered":"2024-03-16T12:19:44.546518+0000","last_clean":"2024-03-16T12:19:44.546518+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:44.546518+0000","last_undegraded":"2024-03-16T12:19:44.546518+0000","last_fullsized":"2024-03-16T12:19:44.546518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548608+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:19:44.548608+0000","last_peered":"2024-03-16T12:19:44.548608+0000","last_clean":"2024-03-16T12:19:44.548608+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:19:44.548608+0000","last_undegraded":"2024-03-16T12:19:44.548608+0000","last_fullsized":"2024-03-16T12:19:44.548608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567444+0000","last_change":"2024-03-16T12:19:44.566987+0000","last_active":"2024-03-16T12:19:44.567444+0000","last_peered":"2024-03-16T12:19:44.567444+0000","last_clean":"2024-03-16T12:19:44.567444+0000","last_became_active":"2024-03-16T12:19:44.566529+0000","last_became_peered":"2024-03-16T12:19:44.566529+0000","last_unstale":"2024-03-16T12:19:44.567444+0000","last_undegraded":"2024-03-16T12:19:44.567444+0000","last_fullsized":"2024-03-16T12:19:44.567444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:53:03.124434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785697+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:20:12.785697+0000","last_peered":"2024-03-16T12:20:12.785697+0000","last_clean":"2024-03-16T12:20:12.785697+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:20:12.785697+0000","last_undegraded":"2024-03-16T12:20:12.785697+0000","last_fullsized":"2024-03-16T12:20:12.785697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548707+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:44.548707+0000","last_peered":"2024-03-16T12:19:44.548707+0000","last_clean":"2024-03-16T12:19:44.548707+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:44.548707+0000","last_undegraded":"2024-03-16T12:19:44.548707+0000","last_fullsized":"2024-03-16T12:19:44.548707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559891+0000","last_change":"2024-03-16T12:19:44.559544+0000","last_active":"2024-03-16T12:19:44.559891+0000","last_peered":"2024-03-16T12:19:44.559891+0000","last_clean":"2024-03-16T12:19:44.559891+0000","last_became_active":"2024-03-16T12:19:44.559030+0000","last_became_peered":"2024-03-16T12:19:44.559030+0000","last_unstale":"2024-03-16T12:19:44.559891+0000","last_undegraded":"2024-03-16T12:19:44.559891+0000","last_fullsized":"2024-03-16T12:19:44.559891+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:46.496512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7,6],"acting":[11,10,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543822+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:44.543822+0000","last_peered":"2024-03-16T12:19:44.543822+0000","last_clean":"2024-03-16T12:19:44.543822+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:44.543822+0000","last_undegraded":"2024-03-16T12:19:44.543822+0000","last_fullsized":"2024-03-16T12:19:44.543822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787665+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:20:12.787665+0000","last_peered":"2024-03-16T12:20:12.787665+0000","last_clean":"2024-03-16T12:20:12.787665+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:20:12.787665+0000","last_undegraded":"2024-03-16T12:20:12.787665+0000","last_fullsized":"2024-03-16T12:20:12.787665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546514+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:19:44.546514+0000","last_peered":"2024-03-16T12:19:44.546514+0000","last_clean":"2024-03-16T12:19:44.546514+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:19:44.546514+0000","last_undegraded":"2024-03-16T12:19:44.546514+0000","last_fullsized":"2024-03-16T12:19:44.546514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:50.091668+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:20:50.091668+0000","last_peered":"2024-03-16T12:20:50.091668+0000","last_clean":"2024-03-16T12:20:50.091668+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:20:50.091668+0000","last_undegraded":"2024-03-16T12:20:50.091668+0000","last_fullsized":"2024-03-16T12:20:50.091668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788697+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:20:12.788697+0000","last_peered":"2024-03-16T12:20:12.788697+0000","last_clean":"2024-03-16T12:20:12.788697+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:20:12.788697+0000","last_undegraded":"2024-03-16T12:20:12.788697+0000","last_fullsized":"2024-03-16T12:20:12.788697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546828+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:19:44.546828+0000","last_peered":"2024-03-16T12:19:44.546828+0000","last_clean":"2024-03-16T12:19:44.546828+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:19:44.546828+0000","last_undegraded":"2024-03-16T12:19:44.546828+0000","last_fullsized":"2024-03-16T12:19:44.546828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546979+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:44.546979+0000","last_peered":"2024-03-16T12:19:44.546979+0000","last_clean":"2024-03-16T12:19:44.546979+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:44.546979+0000","last_undegraded":"2024-03-16T12:19:44.546979+0000","last_fullsized":"2024-03-16T12:19:44.546979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804936+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:20:12.804936+0000","last_peered":"2024-03-16T12:20:12.804936+0000","last_clean":"2024-03-16T12:20:12.804936+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:20:12.804936+0000","last_undegraded":"2024-03-16T12:20:12.804936+0000","last_fullsized":"2024-03-16T12:20:12.804936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546486+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:19:44.546486+0000","last_peered":"2024-03-16T12:19:44.546486+0000","last_clean":"2024-03-16T12:19:44.546486+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:19:44.546486+0000","last_undegraded":"2024-03-16T12:19:44.546486+0000","last_fullsized":"2024-03-16T12:19:44.546486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168879+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:45.168879+0000","last_peered":"2024-03-16T12:19:45.168879+0000","last_clean":"2024-03-16T12:19:45.168879+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:45.168879+0000","last_undegraded":"2024-03-16T12:19:45.168879+0000","last_fullsized":"2024-03-16T12:19:45.168879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.748250+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:20:12.748250+0000","last_peered":"2024-03-16T12:20:12.748250+0000","last_clean":"2024-03-16T12:20:12.748250+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:20:12.748250+0000","last_undegraded":"2024-03-16T12:20:12.748250+0000","last_fullsized":"2024-03-16T12:20:12.748250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847167+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:19:44.847167+0000","last_peered":"2024-03-16T12:19:44.847167+0000","last_clean":"2024-03-16T12:19:44.847167+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:19:44.847167+0000","last_undegraded":"2024-03-16T12:19:44.847167+0000","last_fullsized":"2024-03-16T12:19:44.847167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546586+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:44.546586+0000","last_peered":"2024-03-16T12:19:44.546586+0000","last_clean":"2024-03-16T12:19:44.546586+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:44.546586+0000","last_undegraded":"2024-03-16T12:19:44.546586+0000","last_fullsized":"2024-03-16T12:19:44.546586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803505+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:20:12.803505+0000","last_peered":"2024-03-16T12:20:12.803505+0000","last_clean":"2024-03-16T12:20:12.803505+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:20:12.803505+0000","last_undegraded":"2024-03-16T12:20:12.803505+0000","last_fullsized":"2024-03-16T12:20:12.803505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546670+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:19:44.546670+0000","last_peered":"2024-03-16T12:19:44.546670+0000","last_clean":"2024-03-16T12:19:44.546670+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:19:44.546670+0000","last_undegraded":"2024-03-16T12:19:44.546670+0000","last_fullsized":"2024-03-16T12:19:44.546670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547065+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:19:44.547065+0000","last_peered":"2024-03-16T12:19:44.547065+0000","last_clean":"2024-03-16T12:19:44.547065+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:19:44.547065+0000","last_undegraded":"2024-03-16T12:19:44.547065+0000","last_fullsized":"2024-03-16T12:19:44.547065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805338+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:20:12.805338+0000","last_peered":"2024-03-16T12:20:12.805338+0000","last_clean":"2024-03-16T12:20:12.805338+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:20:12.805338+0000","last_undegraded":"2024-03-16T12:20:12.805338+0000","last_fullsized":"2024-03-16T12:20:12.805338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546343+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:19:44.546343+0000","last_peered":"2024-03-16T12:19:44.546343+0000","last_clean":"2024-03-16T12:19:44.546343+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:19:44.546343+0000","last_undegraded":"2024-03-16T12:19:44.546343+0000","last_fullsized":"2024-03-16T12:19:44.546343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555974+0000","last_change":"2024-03-16T12:19:44.555974+0000","last_active":"2024-03-16T12:19:44.555974+0000","last_peered":"2024-03-16T12:19:44.555974+0000","last_clean":"2024-03-16T12:19:44.555974+0000","last_became_active":"2024-03-16T12:19:44.550254+0000","last_became_peered":"2024-03-16T12:19:44.550254+0000","last_unstale":"2024-03-16T12:19:44.555974+0000","last_undegraded":"2024-03-16T12:19:44.555974+0000","last_fullsized":"2024-03-16T12:19:44.555974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789538+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:20:12.789538+0000","last_peered":"2024-03-16T12:20:12.789538+0000","last_clean":"2024-03-16T12:20:12.789538+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:20:12.789538+0000","last_undegraded":"2024-03-16T12:20:12.789538+0000","last_fullsized":"2024-03-16T12:20:12.789538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168880+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:19:45.168880+0000","last_peered":"2024-03-16T12:19:45.168880+0000","last_clean":"2024-03-16T12:19:45.168880+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:19:45.168880+0000","last_undegraded":"2024-03-16T12:19:45.168880+0000","last_fullsized":"2024-03-16T12:19:45.168880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548184+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:19:44.548184+0000","last_peered":"2024-03-16T12:19:44.548184+0000","last_clean":"2024-03-16T12:19:44.548184+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:19:44.548184+0000","last_undegraded":"2024-03-16T12:19:44.548184+0000","last_fullsized":"2024-03-16T12:19:44.548184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786719+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:20:12.786719+0000","last_peered":"2024-03-16T12:20:12.786719+0000","last_clean":"2024-03-16T12:20:12.786719+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:20:12.786719+0000","last_undegraded":"2024-03-16T12:20:12.786719+0000","last_fullsized":"2024-03-16T12:20:12.786719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172049+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:19:45.172049+0000","last_peered":"2024-03-16T12:19:45.172049+0000","last_clean":"2024-03-16T12:19:45.172049+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:19:45.172049+0000","last_undegraded":"2024-03-16T12:19:45.172049+0000","last_fullsized":"2024-03-16T12:19:45.172049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550000+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:19:44.550000+0000","last_peered":"2024-03-16T12:19:44.550000+0000","last_clean":"2024-03-16T12:19:44.550000+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:19:44.550000+0000","last_undegraded":"2024-03-16T12:19:44.550000+0000","last_fullsized":"2024-03-16T12:19:44.550000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791712+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:20:12.791712+0000","last_peered":"2024-03-16T12:20:12.791712+0000","last_clean":"2024-03-16T12:20:12.791712+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:20:12.791712+0000","last_undegraded":"2024-03-16T12:20:12.791712+0000","last_fullsized":"2024-03-16T12:20:12.791712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549274+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:19:44.549274+0000","last_peered":"2024-03-16T12:19:44.549274+0000","last_clean":"2024-03-16T12:19:44.549274+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:19:44.549274+0000","last_undegraded":"2024-03-16T12:19:44.549274+0000","last_fullsized":"2024-03-16T12:19:44.549274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559640+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:19:44.559640+0000","last_peered":"2024-03-16T12:19:44.559640+0000","last_clean":"2024-03-16T12:19:44.559640+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:19:44.559640+0000","last_undegraded":"2024-03-16T12:19:44.559640+0000","last_fullsized":"2024-03-16T12:19:44.559640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.771744+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:20:12.771744+0000","last_peered":"2024-03-16T12:20:12.771744+0000","last_clean":"2024-03-16T12:20:12.771744+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:20:12.771744+0000","last_undegraded":"2024-03-16T12:20:12.771744+0000","last_fullsized":"2024-03-16T12:20:12.771744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774255+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:20:12.774255+0000","last_peered":"2024-03-16T12:20:12.774255+0000","last_clean":"2024-03-16T12:20:12.774255+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:20:12.774255+0000","last_undegraded":"2024-03-16T12:20:12.774255+0000","last_fullsized":"2024-03-16T12:20:12.774255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781112+0000","last_change":"2024-03-16T12:19:44.847899+0000","last_active":"2024-03-16T12:20:12.781112+0000","last_peered":"2024-03-16T12:20:12.781112+0000","last_clean":"2024-03-16T12:20:12.781112+0000","last_became_active":"2024-03-16T12:19:44.847589+0000","last_became_peered":"2024-03-16T12:19:44.847589+0000","last_unstale":"2024-03-16T12:20:12.781112+0000","last_undegraded":"2024-03-16T12:20:12.781112+0000","last_fullsized":"2024-03-16T12:20:12.781112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:59.091516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.802828+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:20:12.802828+0000","last_peered":"2024-03-16T12:20:12.802828+0000","last_clean":"2024-03-16T12:20:12.802828+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:20:12.802828+0000","last_undegraded":"2024-03-16T12:20:12.802828+0000","last_fullsized":"2024-03-16T12:20:12.802828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804518+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:20:12.804518+0000","last_peered":"2024-03-16T12:20:12.804518+0000","last_clean":"2024-03-16T12:20:12.804518+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:20:12.804518+0000","last_undegraded":"2024-03-16T12:20:12.804518+0000","last_fullsized":"2024-03-16T12:20:12.804518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805439+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:20:12.805439+0000","last_peered":"2024-03-16T12:20:12.805439+0000","last_clean":"2024-03-16T12:20:12.805439+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:20:12.805439+0000","last_undegraded":"2024-03-16T12:20:12.805439+0000","last_fullsized":"2024-03-16T12:20:12.805439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770556+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:20:12.770556+0000","last_peered":"2024-03-16T12:20:12.770556+0000","last_clean":"2024-03-16T12:20:12.770556+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:20:12.770556+0000","last_undegraded":"2024-03-16T12:20:12.770556+0000","last_fullsized":"2024-03-16T12:20:12.770556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776877+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:20:12.776877+0000","last_peered":"2024-03-16T12:20:12.776877+0000","last_clean":"2024-03-16T12:20:12.776877+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:20:12.776877+0000","last_undegraded":"2024-03-16T12:20:12.776877+0000","last_fullsized":"2024-03-16T12:20:12.776877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.777318+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:20:12.777318+0000","last_peered":"2024-03-16T12:20:12.777318+0000","last_clean":"2024-03-16T12:20:12.777318+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:20:12.777318+0000","last_undegraded":"2024-03-16T12:20:12.777318+0000","last_fullsized":"2024-03-16T12:20:12.777318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799957+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:20:12.799957+0000","last_peered":"2024-03-16T12:20:12.799957+0000","last_clean":"2024-03-16T12:20:12.799957+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:20:12.799957+0000","last_undegraded":"2024-03-16T12:20:12.799957+0000","last_fullsized":"2024-03-16T12:20:12.799957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.760742+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:20:12.760742+0000","last_peered":"2024-03-16T12:20:12.760742+0000","last_clean":"2024-03-16T12:20:12.760742+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:20:12.760742+0000","last_undegraded":"2024-03-16T12:20:12.760742+0000","last_fullsized":"2024-03-16T12:20:12.760742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800264+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:20:12.800264+0000","last_peered":"2024-03-16T12:20:12.800264+0000","last_clean":"2024-03-16T12:20:12.800264+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:20:12.800264+0000","last_undegraded":"2024-03-16T12:20:12.800264+0000","last_fullsized":"2024-03-16T12:20:12.800264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.806609+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:20:12.806609+0000","last_peered":"2024-03-16T12:20:12.806609+0000","last_clean":"2024-03-16T12:20:12.806609+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:20:12.806609+0000","last_undegraded":"2024-03-16T12:20:12.806609+0000","last_fullsized":"2024-03-16T12:20:12.806609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781562+0000","last_change":"2024-03-16T12:19:44.849288+0000","last_active":"2024-03-16T12:20:12.781562+0000","last_peered":"2024-03-16T12:20:12.781562+0000","last_clean":"2024-03-16T12:20:12.781562+0000","last_became_active":"2024-03-16T12:19:44.848869+0000","last_became_peered":"2024-03-16T12:19:44.848869+0000","last_unstale":"2024-03-16T12:20:12.781562+0000","last_undegraded":"2024-03-16T12:20:12.781562+0000","last_fullsized":"2024-03-16T12:20:12.781562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:36.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72185,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":929,"num_read_kb":541,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":216555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":500849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28696,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716456,"statfs":{"total":95995035648,"available":95965650944,"internally_reserved":0,"allocated":1720320,"data_stored":1156658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":500971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1096799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495693,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":564402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":565718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":501187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1092077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:21:39.848 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-16T12:21:39.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T12:21:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 -- 172.21.15.98:0/3225722869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 msgr2=0x7fe8600765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 --2- 172.21.15.98:0/3225722869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe8600765c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe848009900 tx=0x7fe84802bc50 comp rx=0 tx=0).stop 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 -- 172.21.15.98:0/3225722869 shutdown_connections 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 --2- 172.21.15.98:0/3225722869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe860077d80 0x7fe860078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 --2- 172.21.15.98:0/3225722869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8600773e0 0x7fe860077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 --2- 172.21.15.98:0/3225722869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe8600765c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 -- 172.21.15.98:0/3225722869 >> 172.21.15.98:0/3225722869 conn(0x7fe8600fef90 msgr2=0x7fe8601013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 -- 172.21.15.98:0/3225722869 shutdown_connections 2024-03-16T12:21:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.118+0000 7fe865800640 1 -- 172.21.15.98:0/3225722869 wait complete. 2024-03-16T12:21:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.119+0000 7fe865800640 1 Processor -- start 2024-03-16T12:21:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.119+0000 7fe865800640 1 -- start start 2024-03-16T12:21:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.119+0000 7fe865800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.119+0000 7fe865800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 0x7fe86019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.119+0000 7fe865800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860077d80 0x7fe8601a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe865800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe860113a90 con 0x7fe860077d80 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe865800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe860113910 con 0x7fe8600773e0 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe865800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe860113c10 con 0x7fe8600761a0 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56226/0 (socket says 172.21.15.98:56226) 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 -- 172.21.15.98:0/925508374 learned_addr learned my addr 172.21.15.98:0/925508374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe857e00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 0x7fe86019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85f400640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860077d80 0x7fe8601a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 -- 172.21.15.98:0/925508374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 msgr2=0x7fe86019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 0x7fe86019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 -- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860077d80 msgr2=0x7fe8601a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860077d80 0x7fe8601a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe85ea00640 1 -- 172.21.15.98:0/925508374 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8601a5d00 con 0x7fe8600761a0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.120+0000 7fe857e00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 0x7fe86019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe85ea00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe84802c260 tx=0x7fe84802c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe848041070 con 0x7fe8600761a0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe848004450 con 0x7fe8600761a0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8601a5f90 con 0x7fe8600761a0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe848034440 con 0x7fe8600761a0 2024-03-16T12:21:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.121+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8601a6400 con 0x7fe8600761a0 2024-03-16T12:21:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.122+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe860110370 con 0x7fe8600761a0 2024-03-16T12:21:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.124+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe848044070 con 0x7fe8600761a0 2024-03-16T12:21:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.125+0000 7fe857400640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 0x7fe8440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.125+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8480f08a0 con 0x7fe8600761a0 2024-03-16T12:21:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.125+0000 7fe857e00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 0x7fe8440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.126+0000 7fe857e00640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 0x7fe8440a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe86019f5f0 tx=0x7fe850005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.129+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe8480bcf30 con 0x7fe8600761a0 2024-03-16T12:21:40.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:39 smithi098 ceph-mon[31633]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:40.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:39 smithi153 ceph-mon[36901]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:40.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.327+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 --> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe860061c20 con 0x7fe8440a2800 2024-03-16T12:21:40.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:39 smithi173 ceph-mon[36632]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.339+0000 7fe857400640 1 -- 172.21.15.98:0/925508374 <== mgr.14152 v2:172.21.15.98:6800/3089653509 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340494 (secure 0 0 0) 0x7fe860061c20 con 0x7fe8440a2800 2024-03-16T12:21:40.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 msgr2=0x7fe8440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 0x7fe8440a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe86019f5f0 tx=0x7fe850005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 msgr2=0x7fe86019e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe84802c260 tx=0x7fe84802c670 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 shutdown_connections 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fe8440a2800 0x7fe8440a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860077d80 0x7fe8601a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8600773e0 0x7fe86019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.344+0000 7fe865800640 1 --2- 172.21.15.98:0/925508374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8600761a0 0x7fe86019e0e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.345+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 >> 172.21.15.98:0/925508374 conn(0x7fe8600fef90 msgr2=0x7fe860100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.345+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 shutdown_connections 2024-03-16T12:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.345+0000 7fe865800640 1 -- 172.21.15.98:0/925508374 wait complete. 2024-03-16T12:21:40.345 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-16T12:21:38.851289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662787,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1055,"num_read_kb":650,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384060,"kb_used_data":14108,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557764,"statfs":{"total":1151940427776,"available":1151547150336,"internally_reserved":0,"allocated":14446592,"data_stored":7974071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5993,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":43,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004627"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803803+0000","last_change":"2024-03-16T12:18:40.521636+0000","last_active":"2024-03-16T12:20:12.803803+0000","last_peered":"2024-03-16T12:20:12.803803+0000","last_clean":"2024-03-16T12:20:12.803803+0000","last_became_active":"2024-03-16T12:18:40.521268+0000","last_became_peered":"2024-03-16T12:18:40.521268+0000","last_unstale":"2024-03-16T12:20:12.803803+0000","last_undegraded":"2024-03-16T12:20:12.803803+0000","last_fullsized":"2024-03-16T12:20:12.803803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:36.227580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798435+0000","last_change":"2024-03-16T12:18:40.514757+0000","last_active":"2024-03-16T12:20:12.798435+0000","last_peered":"2024-03-16T12:20:12.798435+0000","last_clean":"2024-03-16T12:20:12.798435+0000","last_became_active":"2024-03-16T12:18:40.514425+0000","last_became_peered":"2024-03-16T12:18:40.514425+0000","last_unstale":"2024-03-16T12:20:12.798435+0000","last_undegraded":"2024-03-16T12:20:12.798435+0000","last_fullsized":"2024-03-16T12:20:12.798435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:52:17.246042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800413+0000","last_change":"2024-03-16T12:18:40.517567+0000","last_active":"2024-03-16T12:20:12.800413+0000","last_peered":"2024-03-16T12:20:12.800413+0000","last_clean":"2024-03-16T12:20:12.800413+0000","last_became_active":"2024-03-16T12:18:40.511800+0000","last_became_peered":"2024-03-16T12:18:40.511800+0000","last_unstale":"2024-03-16T12:20:12.800413+0000","last_undegraded":"2024-03-16T12:20:12.800413+0000","last_fullsized":"2024-03-16T12:20:12.800413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:44:35.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755748+0000","last_change":"2024-03-16T12:18:40.504304+0000","last_active":"2024-03-16T12:20:12.755748+0000","last_peered":"2024-03-16T12:20:12.755748+0000","last_clean":"2024-03-16T12:20:12.755748+0000","last_became_active":"2024-03-16T12:18:40.502794+0000","last_became_peered":"2024-03-16T12:18:40.502794+0000","last_unstale":"2024-03-16T12:20:12.755748+0000","last_undegraded":"2024-03-16T12:20:12.755748+0000","last_fullsized":"2024-03-16T12:20:12.755748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:14.044961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791944+0000","last_change":"2024-03-16T12:18:40.510368+0000","last_active":"2024-03-16T12:20:12.791944+0000","last_peered":"2024-03-16T12:20:12.791944+0000","last_clean":"2024-03-16T12:20:12.791944+0000","last_became_active":"2024-03-16T12:18:40.509984+0000","last_became_peered":"2024-03-16T12:18:40.509984+0000","last_unstale":"2024-03-16T12:20:12.791944+0000","last_undegraded":"2024-03-16T12:20:12.791944+0000","last_fullsized":"2024-03-16T12:20:12.791944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:15.599303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785928+0000","last_change":"2024-03-16T12:18:40.512686+0000","last_active":"2024-03-16T12:20:12.785928+0000","last_peered":"2024-03-16T12:20:12.785928+0000","last_clean":"2024-03-16T12:20:12.785928+0000","last_became_active":"2024-03-16T12:18:40.511490+0000","last_became_peered":"2024-03-16T12:18:40.511490+0000","last_unstale":"2024-03-16T12:20:12.785928+0000","last_undegraded":"2024-03-16T12:20:12.785928+0000","last_fullsized":"2024-03-16T12:20:12.785928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:25.155092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.773384+0000","last_change":"2024-03-16T12:18:40.511487+0000","last_active":"2024-03-16T12:20:12.773384+0000","last_peered":"2024-03-16T12:20:12.773384+0000","last_clean":"2024-03-16T12:20:12.773384+0000","last_became_active":"2024-03-16T12:18:40.511116+0000","last_became_peered":"2024-03-16T12:18:40.511116+0000","last_unstale":"2024-03-16T12:20:12.773384+0000","last_undegraded":"2024-03-16T12:20:12.773384+0000","last_fullsized":"2024-03-16T12:20:12.773384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:50:25.557227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793133+0000","last_change":"2024-03-16T12:18:40.522528+0000","last_active":"2024-03-16T12:20:12.793133+0000","last_peered":"2024-03-16T12:20:12.793133+0000","last_clean":"2024-03-16T12:20:12.793133+0000","last_became_active":"2024-03-16T12:18:40.522044+0000","last_became_peered":"2024-03-16T12:18:40.522044+0000","last_unstale":"2024-03-16T12:20:12.793133+0000","last_undegraded":"2024-03-16T12:20:12.793133+0000","last_fullsized":"2024-03-16T12:20:12.793133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:37.603448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785095+0000","last_change":"2024-03-16T12:18:40.505368+0000","last_active":"2024-03-16T12:20:12.785095+0000","last_peered":"2024-03-16T12:20:12.785095+0000","last_clean":"2024-03-16T12:20:12.785095+0000","last_became_active":"2024-03-16T12:18:40.505008+0000","last_became_peered":"2024-03-16T12:18:40.505008+0000","last_unstale":"2024-03-16T12:20:12.785095+0000","last_undegraded":"2024-03-16T12:20:12.785095+0000","last_fullsized":"2024-03-16T12:20:12.785095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:40.422634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785364+0000","last_change":"2024-03-16T12:19:44.848697+0000","last_active":"2024-03-16T12:20:12.785364+0000","last_peered":"2024-03-16T12:20:12.785364+0000","last_clean":"2024-03-16T12:20:12.785364+0000","last_became_active":"2024-03-16T12:19:44.848414+0000","last_became_peered":"2024-03-16T12:19:44.848414+0000","last_unstale":"2024-03-16T12:20:12.785364+0000","last_undegraded":"2024-03-16T12:20:12.785364+0000","last_fullsized":"2024-03-16T12:20:12.785364+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:04:54.721056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786403+0000","last_change":"2024-03-16T12:18:40.504496+0000","last_active":"2024-03-16T12:20:12.786403+0000","last_peered":"2024-03-16T12:20:12.786403+0000","last_clean":"2024-03-16T12:20:12.786403+0000","last_became_active":"2024-03-16T12:18:40.504010+0000","last_became_peered":"2024-03-16T12:18:40.504010+0000","last_unstale":"2024-03-16T12:20:12.786403+0000","last_undegraded":"2024-03-16T12:20:12.786403+0000","last_fullsized":"2024-03-16T12:20:12.786403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:00.940946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755018+0000","last_change":"2024-03-16T12:18:40.500267+0000","last_active":"2024-03-16T12:20:12.755018+0000","last_peered":"2024-03-16T12:20:12.755018+0000","last_clean":"2024-03-16T12:20:12.755018+0000","last_became_active":"2024-03-16T12:18:40.499904+0000","last_became_peered":"2024-03-16T12:18:40.499904+0000","last_unstale":"2024-03-16T12:20:12.755018+0000","last_undegraded":"2024-03-16T12:20:12.755018+0000","last_fullsized":"2024-03-16T12:20:12.755018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:20.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790423+0000","last_change":"2024-03-16T12:18:40.503136+0000","last_active":"2024-03-16T12:20:12.790423+0000","last_peered":"2024-03-16T12:20:12.790423+0000","last_clean":"2024-03-16T12:20:12.790423+0000","last_became_active":"2024-03-16T12:18:40.502725+0000","last_became_peered":"2024-03-16T12:18:40.502725+0000","last_unstale":"2024-03-16T12:20:12.790423+0000","last_undegraded":"2024-03-16T12:20:12.790423+0000","last_fullsized":"2024-03-16T12:20:12.790423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:32:40.565823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794049+0000","last_change":"2024-03-16T12:18:40.513710+0000","last_active":"2024-03-16T12:20:12.794049+0000","last_peered":"2024-03-16T12:20:12.794049+0000","last_clean":"2024-03-16T12:20:12.794049+0000","last_became_active":"2024-03-16T12:18:40.513375+0000","last_became_peered":"2024-03-16T12:18:40.513375+0000","last_unstale":"2024-03-16T12:20:12.794049+0000","last_undegraded":"2024-03-16T12:20:12.794049+0000","last_fullsized":"2024-03-16T12:20:12.794049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:08.487694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.795950+0000","last_change":"2024-03-16T12:18:40.513764+0000","last_active":"2024-03-16T12:20:12.795950+0000","last_peered":"2024-03-16T12:20:12.795950+0000","last_clean":"2024-03-16T12:20:12.795950+0000","last_became_active":"2024-03-16T12:18:40.513478+0000","last_became_peered":"2024-03-16T12:18:40.513478+0000","last_unstale":"2024-03-16T12:20:12.795950+0000","last_undegraded":"2024-03-16T12:20:12.795950+0000","last_fullsized":"2024-03-16T12:20:12.795950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:21:06.745042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803918+0000","last_change":"2024-03-16T12:18:40.517461+0000","last_active":"2024-03-16T12:20:12.803918+0000","last_peered":"2024-03-16T12:20:12.803918+0000","last_clean":"2024-03-16T12:20:12.803918+0000","last_became_active":"2024-03-16T12:18:40.517131+0000","last_became_peered":"2024-03-16T12:18:40.517131+0000","last_unstale":"2024-03-16T12:20:12.803918+0000","last_undegraded":"2024-03-16T12:20:12.803918+0000","last_fullsized":"2024-03-16T12:20:12.803918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:59:09.789619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804028+0000","last_change":"2024-03-16T12:18:40.521710+0000","last_active":"2024-03-16T12:20:12.804028+0000","last_peered":"2024-03-16T12:20:12.804028+0000","last_clean":"2024-03-16T12:20:12.804028+0000","last_became_active":"2024-03-16T12:18:40.521282+0000","last_became_peered":"2024-03-16T12:18:40.521282+0000","last_unstale":"2024-03-16T12:20:12.804028+0000","last_undegraded":"2024-03-16T12:20:12.804028+0000","last_fullsized":"2024-03-16T12:20:12.804028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:38:13.644866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786205+0000","last_change":"2024-03-16T12:18:40.517509+0000","last_active":"2024-03-16T12:20:12.786205+0000","last_peered":"2024-03-16T12:20:12.786205+0000","last_clean":"2024-03-16T12:20:12.786205+0000","last_became_active":"2024-03-16T12:18:40.517015+0000","last_became_peered":"2024-03-16T12:18:40.517015+0000","last_unstale":"2024-03-16T12:20:12.786205+0000","last_undegraded":"2024-03-16T12:20:12.786205+0000","last_fullsized":"2024-03-16T12:20:12.786205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:38.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792899+0000","last_change":"2024-03-16T12:18:40.522094+0000","last_active":"2024-03-16T12:20:12.792899+0000","last_peered":"2024-03-16T12:20:12.792899+0000","last_clean":"2024-03-16T12:20:12.792899+0000","last_became_active":"2024-03-16T12:18:40.521764+0000","last_became_peered":"2024-03-16T12:18:40.521764+0000","last_unstale":"2024-03-16T12:20:12.792899+0000","last_undegraded":"2024-03-16T12:20:12.792899+0000","last_fullsized":"2024-03-16T12:20:12.792899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:57.617571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547296+0000","last_change":"2024-03-16T12:18:44.583892+0000","last_active":"2024-03-16T12:19:44.547296+0000","last_peered":"2024-03-16T12:19:44.547296+0000","last_clean":"2024-03-16T12:19:44.547296+0000","last_became_active":"2024-03-16T12:18:44.583230+0000","last_became_peered":"2024-03-16T12:18:44.583230+0000","last_unstale":"2024-03-16T12:19:44.547296+0000","last_undegraded":"2024-03-16T12:19:44.547296+0000","last_fullsized":"2024-03-16T12:19:44.547296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:29:51.067897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847301+0000","last_change":"2024-03-16T12:18:39.987537+0000","last_active":"2024-03-16T12:19:44.847301+0000","last_peered":"2024-03-16T12:19:44.847301+0000","last_clean":"2024-03-16T12:19:44.847301+0000","last_became_active":"2024-03-16T12:18:39.983108+0000","last_became_peered":"2024-03-16T12:18:39.983108+0000","last_unstale":"2024-03-16T12:19:44.847301+0000","last_undegraded":"2024-03-16T12:19:44.847301+0000","last_fullsized":"2024-03-16T12:19:44.847301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:52.859199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.769143+0000","last_change":"2024-03-16T12:18:40.525937+0000","last_active":"2024-03-16T12:20:12.769143+0000","last_peered":"2024-03-16T12:20:12.769143+0000","last_clean":"2024-03-16T12:20:12.769143+0000","last_became_active":"2024-03-16T12:18:40.523608+0000","last_became_peered":"2024-03-16T12:18:40.523608+0000","last_unstale":"2024-03-16T12:20:12.769143+0000","last_undegraded":"2024-03-16T12:20:12.769143+0000","last_fullsized":"2024-03-16T12:20:12.769143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:23:35.319047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559668+0000","last_change":"2024-03-16T12:19:44.559192+0000","last_active":"2024-03-16T12:19:44.559668+0000","last_peered":"2024-03-16T12:19:44.559668+0000","last_clean":"2024-03-16T12:19:44.559668+0000","last_became_active":"2024-03-16T12:19:44.558795+0000","last_became_peered":"2024-03-16T12:19:44.558795+0000","last_unstale":"2024-03-16T12:19:44.559668+0000","last_undegraded":"2024-03-16T12:19:44.559668+0000","last_fullsized":"2024-03-16T12:19:44.559668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:41:28.267737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.552034+0000","last_change":"2024-03-16T12:18:40.004653+0000","last_active":"2024-03-16T12:19:44.552034+0000","last_peered":"2024-03-16T12:19:44.552034+0000","last_clean":"2024-03-16T12:19:44.552034+0000","last_became_active":"2024-03-16T12:18:40.003274+0000","last_became_peered":"2024-03-16T12:18:40.003274+0000","last_unstale":"2024-03-16T12:19:44.552034+0000","last_undegraded":"2024-03-16T12:19:44.552034+0000","last_fullsized":"2024-03-16T12:19:44.552034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:26.438724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804064+0000","last_change":"2024-03-16T12:18:40.519496+0000","last_active":"2024-03-16T12:20:12.804064+0000","last_peered":"2024-03-16T12:20:12.804064+0000","last_clean":"2024-03-16T12:20:12.804064+0000","last_became_active":"2024-03-16T12:18:40.519083+0000","last_became_peered":"2024-03-16T12:18:40.519083+0000","last_unstale":"2024-03-16T12:20:12.804064+0000","last_undegraded":"2024-03-16T12:20:12.804064+0000","last_fullsized":"2024-03-16T12:20:12.804064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:33.488006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546380+0000","last_change":"2024-03-16T12:18:44.587738+0000","last_active":"2024-03-16T12:19:44.546380+0000","last_peered":"2024-03-16T12:19:44.546380+0000","last_clean":"2024-03-16T12:19:44.546380+0000","last_became_active":"2024-03-16T12:18:44.587349+0000","last_became_peered":"2024-03-16T12:18:44.587349+0000","last_unstale":"2024-03-16T12:19:44.546380+0000","last_undegraded":"2024-03-16T12:19:44.546380+0000","last_fullsized":"2024-03-16T12:19:44.546380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:42.808007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543851+0000","last_change":"2024-03-16T12:18:39.981145+0000","last_active":"2024-03-16T12:19:44.543851+0000","last_peered":"2024-03-16T12:19:44.543851+0000","last_clean":"2024-03-16T12:19:44.543851+0000","last_became_active":"2024-03-16T12:18:39.980322+0000","last_became_peered":"2024-03-16T12:18:39.980322+0000","last_unstale":"2024-03-16T12:19:44.543851+0000","last_undegraded":"2024-03-16T12:19:44.543851+0000","last_fullsized":"2024-03-16T12:19:44.543851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:12.204417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789084+0000","last_change":"2024-03-16T12:18:40.507325+0000","last_active":"2024-03-16T12:20:12.789084+0000","last_peered":"2024-03-16T12:20:12.789084+0000","last_clean":"2024-03-16T12:20:12.789084+0000","last_became_active":"2024-03-16T12:18:40.506992+0000","last_became_peered":"2024-03-16T12:18:40.506992+0000","last_unstale":"2024-03-16T12:20:12.789084+0000","last_undegraded":"2024-03-16T12:20:12.789084+0000","last_fullsized":"2024-03-16T12:20:12.789084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:49.560963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546007+0000","last_change":"2024-03-16T12:18:44.580668+0000","last_active":"2024-03-16T12:19:44.546007+0000","last_peered":"2024-03-16T12:19:44.546007+0000","last_clean":"2024-03-16T12:19:44.546007+0000","last_became_active":"2024-03-16T12:18:44.580237+0000","last_became_peered":"2024-03-16T12:18:44.580237+0000","last_unstale":"2024-03-16T12:19:44.546007+0000","last_undegraded":"2024-03-16T12:19:44.546007+0000","last_fullsized":"2024-03-16T12:19:44.546007+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:30:47.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543938+0000","last_change":"2024-03-16T12:18:39.979793+0000","last_active":"2024-03-16T12:19:44.543938+0000","last_peered":"2024-03-16T12:19:44.543938+0000","last_clean":"2024-03-16T12:19:44.543938+0000","last_became_active":"2024-03-16T12:18:39.979393+0000","last_became_peered":"2024-03-16T12:18:39.979393+0000","last_unstale":"2024-03-16T12:19:44.543938+0000","last_undegraded":"2024-03-16T12:19:44.543938+0000","last_fullsized":"2024-03-16T12:19:44.543938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:51.637622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.784812+0000","last_change":"2024-03-16T12:18:40.514817+0000","last_active":"2024-03-16T12:20:12.784812+0000","last_peered":"2024-03-16T12:20:12.784812+0000","last_clean":"2024-03-16T12:20:12.784812+0000","last_became_active":"2024-03-16T12:18:40.514397+0000","last_became_peered":"2024-03-16T12:18:40.514397+0000","last_unstale":"2024-03-16T12:20:12.784812+0000","last_undegraded":"2024-03-16T12:20:12.784812+0000","last_fullsized":"2024-03-16T12:20:12.784812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:27:06.334542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547794+0000","last_change":"2024-03-16T12:18:44.570247+0000","last_active":"2024-03-16T12:19:44.547794+0000","last_peered":"2024-03-16T12:19:44.547794+0000","last_clean":"2024-03-16T12:19:44.547794+0000","last_became_active":"2024-03-16T12:18:44.569780+0000","last_became_peered":"2024-03-16T12:18:44.569780+0000","last_unstale":"2024-03-16T12:19:44.547794+0000","last_undegraded":"2024-03-16T12:19:44.547794+0000","last_fullsized":"2024-03-16T12:19:44.547794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:54:03.531637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":187,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.644775+0000","last_change":"2024-03-16T12:18:39.981766+0000","last_active":"2024-03-16T12:21:37.644775+0000","last_peered":"2024-03-16T12:21:37.644775+0000","last_clean":"2024-03-16T12:21:37.644775+0000","last_became_active":"2024-03-16T12:18:39.981417+0000","last_became_peered":"2024-03-16T12:18:39.981417+0000","last_unstale":"2024-03-16T12:21:37.644775+0000","last_undegraded":"2024-03-16T12:21:37.644775+0000","last_fullsized":"2024-03-16T12:21:37.644775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:11.829786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781031+0000","last_change":"2024-03-16T12:18:40.507464+0000","last_active":"2024-03-16T12:20:12.781031+0000","last_peered":"2024-03-16T12:20:12.781031+0000","last_clean":"2024-03-16T12:20:12.781031+0000","last_became_active":"2024-03-16T12:18:40.507140+0000","last_became_peered":"2024-03-16T12:18:40.507140+0000","last_unstale":"2024-03-16T12:20:12.781031+0000","last_undegraded":"2024-03-16T12:20:12.781031+0000","last_fullsized":"2024-03-16T12:20:12.781031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:07.390445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548011+0000","last_change":"2024-03-16T12:18:44.582199+0000","last_active":"2024-03-16T12:19:44.548011+0000","last_peered":"2024-03-16T12:19:44.548011+0000","last_clean":"2024-03-16T12:19:44.548011+0000","last_became_active":"2024-03-16T12:18:44.581775+0000","last_became_peered":"2024-03-16T12:18:44.581775+0000","last_unstale":"2024-03-16T12:19:44.548011+0000","last_undegraded":"2024-03-16T12:19:44.548011+0000","last_fullsized":"2024-03-16T12:19:44.548011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:22:30.364853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547430+0000","last_change":"2024-03-16T12:18:39.987876+0000","last_active":"2024-03-16T12:19:44.547430+0000","last_peered":"2024-03-16T12:19:44.547430+0000","last_clean":"2024-03-16T12:19:44.547430+0000","last_became_active":"2024-03-16T12:18:39.987488+0000","last_became_peered":"2024-03-16T12:18:39.987488+0000","last_unstale":"2024-03-16T12:19:44.547430+0000","last_undegraded":"2024-03-16T12:19:44.547430+0000","last_fullsized":"2024-03-16T12:19:44.547430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:43.387657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.798746+0000","last_change":"2024-03-16T12:18:40.518998+0000","last_active":"2024-03-16T12:20:12.798746+0000","last_peered":"2024-03-16T12:20:12.798746+0000","last_clean":"2024-03-16T12:20:12.798746+0000","last_became_active":"2024-03-16T12:18:40.518685+0000","last_became_peered":"2024-03-16T12:18:40.518685+0000","last_unstale":"2024-03-16T12:20:12.798746+0000","last_undegraded":"2024-03-16T12:20:12.798746+0000","last_fullsized":"2024-03-16T12:20:12.798746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:59:36.220718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545612+0000","last_change":"2024-03-16T12:18:44.591238+0000","last_active":"2024-03-16T12:19:44.545612+0000","last_peered":"2024-03-16T12:19:44.545612+0000","last_clean":"2024-03-16T12:19:44.545612+0000","last_became_active":"2024-03-16T12:18:44.590793+0000","last_became_peered":"2024-03-16T12:18:44.590793+0000","last_unstale":"2024-03-16T12:19:44.545612+0000","last_undegraded":"2024-03-16T12:19:44.545612+0000","last_fullsized":"2024-03-16T12:19:44.545612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:41.102509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555984+0000","last_change":"2024-03-16T12:18:39.983009+0000","last_active":"2024-03-16T12:19:44.555984+0000","last_peered":"2024-03-16T12:19:44.555984+0000","last_clean":"2024-03-16T12:19:44.555984+0000","last_became_active":"2024-03-16T12:18:39.982646+0000","last_became_peered":"2024-03-16T12:18:39.982646+0000","last_unstale":"2024-03-16T12:19:44.555984+0000","last_undegraded":"2024-03-16T12:19:44.555984+0000","last_fullsized":"2024-03-16T12:19:44.555984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:10.110075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.792926+0000","last_change":"2024-03-16T12:18:40.527575+0000","last_active":"2024-03-16T12:20:12.792926+0000","last_peered":"2024-03-16T12:20:12.792926+0000","last_clean":"2024-03-16T12:20:12.792926+0000","last_became_active":"2024-03-16T12:18:40.526414+0000","last_became_peered":"2024-03-16T12:18:40.526414+0000","last_unstale":"2024-03-16T12:20:12.792926+0000","last_undegraded":"2024-03-16T12:20:12.792926+0000","last_fullsized":"2024-03-16T12:20:12.792926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:04:18.267530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172396+0000","last_change":"2024-03-16T12:18:44.576733+0000","last_active":"2024-03-16T12:19:45.172396+0000","last_peered":"2024-03-16T12:19:45.172396+0000","last_clean":"2024-03-16T12:19:45.172396+0000","last_became_active":"2024-03-16T12:18:44.573338+0000","last_became_peered":"2024-03-16T12:18:44.573338+0000","last_unstale":"2024-03-16T12:19:45.172396+0000","last_undegraded":"2024-03-16T12:19:45.172396+0000","last_fullsized":"2024-03-16T12:19:45.172396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:16.483843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549572+0000","last_change":"2024-03-16T12:18:39.979239+0000","last_active":"2024-03-16T12:19:44.549572+0000","last_peered":"2024-03-16T12:19:44.549572+0000","last_clean":"2024-03-16T12:19:44.549572+0000","last_became_active":"2024-03-16T12:18:39.978839+0000","last_became_peered":"2024-03-16T12:18:39.978839+0000","last_unstale":"2024-03-16T12:19:44.549572+0000","last_undegraded":"2024-03-16T12:19:44.549572+0000","last_fullsized":"2024-03-16T12:19:44.549572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:59:31.667044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.801975+0000","last_change":"2024-03-16T12:18:40.520017+0000","last_active":"2024-03-16T12:20:12.801975+0000","last_peered":"2024-03-16T12:20:12.801975+0000","last_clean":"2024-03-16T12:20:12.801975+0000","last_became_active":"2024-03-16T12:18:40.519426+0000","last_became_peered":"2024-03-16T12:18:40.519426+0000","last_unstale":"2024-03-16T12:20:12.801975+0000","last_undegraded":"2024-03-16T12:20:12.801975+0000","last_fullsized":"2024-03-16T12:20:12.801975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:23.554749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546523+0000","last_change":"2024-03-16T12:18:44.579615+0000","last_active":"2024-03-16T12:19:44.546523+0000","last_peered":"2024-03-16T12:19:44.546523+0000","last_clean":"2024-03-16T12:19:44.546523+0000","last_became_active":"2024-03-16T12:18:44.579191+0000","last_became_peered":"2024-03-16T12:18:44.579191+0000","last_unstale":"2024-03-16T12:19:44.546523+0000","last_undegraded":"2024-03-16T12:19:44.546523+0000","last_fullsized":"2024-03-16T12:19:44.546523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:59:39.465517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":181,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:35.641987+0000","last_change":"2024-03-16T12:18:39.983409+0000","last_active":"2024-03-16T12:21:35.641987+0000","last_peered":"2024-03-16T12:21:35.641987+0000","last_clean":"2024-03-16T12:21:35.641987+0000","last_became_active":"2024-03-16T12:18:39.983164+0000","last_became_peered":"2024-03-16T12:18:39.983164+0000","last_unstale":"2024-03-16T12:21:35.641987+0000","last_undegraded":"2024-03-16T12:21:35.641987+0000","last_fullsized":"2024-03-16T12:21:35.641987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:26.006455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787423+0000","last_change":"2024-03-16T12:18:40.512652+0000","last_active":"2024-03-16T12:20:12.787423+0000","last_peered":"2024-03-16T12:20:12.787423+0000","last_clean":"2024-03-16T12:20:12.787423+0000","last_became_active":"2024-03-16T12:18:40.512337+0000","last_became_peered":"2024-03-16T12:18:40.512337+0000","last_unstale":"2024-03-16T12:20:12.787423+0000","last_undegraded":"2024-03-16T12:20:12.787423+0000","last_fullsized":"2024-03-16T12:20:12.787423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:13:05.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845673+0000","last_change":"2024-03-16T12:18:44.592367+0000","last_active":"2024-03-16T12:19:44.845673+0000","last_peered":"2024-03-16T12:19:44.845673+0000","last_clean":"2024-03-16T12:19:44.845673+0000","last_became_active":"2024-03-16T12:18:44.590194+0000","last_became_peered":"2024-03-16T12:18:44.590194+0000","last_unstale":"2024-03-16T12:19:44.845673+0000","last_undegraded":"2024-03-16T12:19:44.845673+0000","last_fullsized":"2024-03-16T12:19:44.845673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:39.107294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548131+0000","last_change":"2024-03-16T12:18:39.980436+0000","last_active":"2024-03-16T12:19:44.548131+0000","last_peered":"2024-03-16T12:19:44.548131+0000","last_clean":"2024-03-16T12:19:44.548131+0000","last_became_active":"2024-03-16T12:18:39.980115+0000","last_became_peered":"2024-03-16T12:18:39.980115+0000","last_unstale":"2024-03-16T12:19:44.548131+0000","last_undegraded":"2024-03-16T12:19:44.548131+0000","last_fullsized":"2024-03-16T12:19:44.548131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:26:39.800360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.794150+0000","last_change":"2024-03-16T12:18:40.512968+0000","last_active":"2024-03-16T12:20:12.794150+0000","last_peered":"2024-03-16T12:20:12.794150+0000","last_clean":"2024-03-16T12:20:12.794150+0000","last_became_active":"2024-03-16T12:18:40.508842+0000","last_became_peered":"2024-03-16T12:18:40.508842+0000","last_unstale":"2024-03-16T12:20:12.794150+0000","last_undegraded":"2024-03-16T12:20:12.794150+0000","last_fullsized":"2024-03-16T12:20:12.794150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:36.636860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546597+0000","last_change":"2024-03-16T12:18:44.587899+0000","last_active":"2024-03-16T12:19:44.546597+0000","last_peered":"2024-03-16T12:19:44.546597+0000","last_clean":"2024-03-16T12:19:44.546597+0000","last_became_active":"2024-03-16T12:18:44.587133+0000","last_became_peered":"2024-03-16T12:18:44.587133+0000","last_unstale":"2024-03-16T12:19:44.546597+0000","last_undegraded":"2024-03-16T12:19:44.546597+0000","last_fullsized":"2024-03-16T12:19:44.546597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:53:23.368135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549488+0000","last_change":"2024-03-16T12:18:39.993135+0000","last_active":"2024-03-16T12:19:44.549488+0000","last_peered":"2024-03-16T12:19:44.549488+0000","last_clean":"2024-03-16T12:19:44.549488+0000","last_became_active":"2024-03-16T12:18:39.992647+0000","last_became_peered":"2024-03-16T12:18:39.992647+0000","last_unstale":"2024-03-16T12:19:44.549488+0000","last_undegraded":"2024-03-16T12:19:44.549488+0000","last_fullsized":"2024-03-16T12:19:44.549488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:56:38.620430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785886+0000","last_change":"2024-03-16T12:18:40.510731+0000","last_active":"2024-03-16T12:20:12.785886+0000","last_peered":"2024-03-16T12:20:12.785886+0000","last_clean":"2024-03-16T12:20:12.785886+0000","last_became_active":"2024-03-16T12:18:40.510405+0000","last_became_peered":"2024-03-16T12:18:40.510405+0000","last_unstale":"2024-03-16T12:20:12.785886+0000","last_undegraded":"2024-03-16T12:20:12.785886+0000","last_fullsized":"2024-03-16T12:20:12.785886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:49:37.905908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.845685+0000","last_change":"2024-03-16T12:18:44.578249+0000","last_active":"2024-03-16T12:19:44.845685+0000","last_peered":"2024-03-16T12:19:44.845685+0000","last_clean":"2024-03-16T12:19:44.845685+0000","last_became_active":"2024-03-16T12:18:44.577860+0000","last_became_peered":"2024-03-16T12:18:44.577860+0000","last_unstale":"2024-03-16T12:19:44.845685+0000","last_undegraded":"2024-03-16T12:19:44.845685+0000","last_fullsized":"2024-03-16T12:19:44.845685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:20:35.946748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'40","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:36.643509+0000","last_change":"2024-03-16T12:18:39.980293+0000","last_active":"2024-03-16T12:21:36.643509+0000","last_peered":"2024-03-16T12:21:36.643509+0000","last_clean":"2024-03-16T12:21:36.643509+0000","last_became_active":"2024-03-16T12:18:39.979938+0000","last_became_peered":"2024-03-16T12:18:39.979938+0000","last_unstale":"2024-03-16T12:21:36.643509+0000","last_undegraded":"2024-03-16T12:21:36.643509+0000","last_fullsized":"2024-03-16T12:21:36.643509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:53.243169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68265,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":311,"num_read_kb":222,"num_write":41,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.753180+0000","last_change":"2024-03-16T12:18:40.500163+0000","last_active":"2024-03-16T12:20:12.753180+0000","last_peered":"2024-03-16T12:20:12.753180+0000","last_clean":"2024-03-16T12:20:12.753180+0000","last_became_active":"2024-03-16T12:18:40.499822+0000","last_became_peered":"2024-03-16T12:18:40.499822+0000","last_unstale":"2024-03-16T12:20:12.753180+0000","last_undegraded":"2024-03-16T12:20:12.753180+0000","last_fullsized":"2024-03-16T12:20:12.753180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:39.363726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550225+0000","last_change":"2024-03-16T12:18:44.582643+0000","last_active":"2024-03-16T12:19:44.550225+0000","last_peered":"2024-03-16T12:19:44.550225+0000","last_clean":"2024-03-16T12:19:44.550225+0000","last_became_active":"2024-03-16T12:18:44.581054+0000","last_became_peered":"2024-03-16T12:18:44.581054+0000","last_unstale":"2024-03-16T12:19:44.550225+0000","last_undegraded":"2024-03-16T12:19:44.550225+0000","last_fullsized":"2024-03-16T12:19:44.550225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:58.558243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.565318+0000","last_change":"2024-03-16T12:19:44.565318+0000","last_active":"2024-03-16T12:19:44.565318+0000","last_peered":"2024-03-16T12:19:44.565318+0000","last_clean":"2024-03-16T12:19:44.565318+0000","last_became_active":"2024-03-16T12:19:44.551024+0000","last_became_peered":"2024-03-16T12:19:44.551024+0000","last_unstale":"2024-03-16T12:19:44.565318+0000","last_undegraded":"2024-03-16T12:19:44.565318+0000","last_fullsized":"2024-03-16T12:19:44.565318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:15.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800693+0000","last_change":"2024-03-16T12:18:40.524362+0000","last_active":"2024-03-16T12:20:12.800693+0000","last_peered":"2024-03-16T12:20:12.800693+0000","last_clean":"2024-03-16T12:20:12.800693+0000","last_became_active":"2024-03-16T12:18:40.523705+0000","last_became_peered":"2024-03-16T12:18:40.523705+0000","last_unstale":"2024-03-16T12:20:12.800693+0000","last_undegraded":"2024-03-16T12:20:12.800693+0000","last_fullsized":"2024-03-16T12:20:12.800693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:00:44.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.545017+0000","last_change":"2024-03-16T12:18:44.580744+0000","last_active":"2024-03-16T12:19:44.545017+0000","last_peered":"2024-03-16T12:19:44.545017+0000","last_clean":"2024-03-16T12:19:44.545017+0000","last_became_active":"2024-03-16T12:18:44.580272+0000","last_became_peered":"2024-03-16T12:18:44.580272+0000","last_unstale":"2024-03-16T12:19:44.545017+0000","last_undegraded":"2024-03-16T12:19:44.545017+0000","last_fullsized":"2024-03-16T12:19:44.545017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:45:53.415065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549187+0000","last_change":"2024-03-16T12:18:39.985655+0000","last_active":"2024-03-16T12:19:44.549187+0000","last_peered":"2024-03-16T12:19:44.549187+0000","last_clean":"2024-03-16T12:19:44.549187+0000","last_became_active":"2024-03-16T12:18:39.985275+0000","last_became_peered":"2024-03-16T12:18:39.985275+0000","last_unstale":"2024-03-16T12:19:44.549187+0000","last_undegraded":"2024-03-16T12:19:44.549187+0000","last_fullsized":"2024-03-16T12:19:44.549187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:10.265125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790351+0000","last_change":"2024-03-16T12:18:40.508284+0000","last_active":"2024-03-16T12:20:12.790351+0000","last_peered":"2024-03-16T12:20:12.790351+0000","last_clean":"2024-03-16T12:20:12.790351+0000","last_became_active":"2024-03-16T12:18:40.507956+0000","last_became_peered":"2024-03-16T12:18:40.507956+0000","last_unstale":"2024-03-16T12:20:12.790351+0000","last_undegraded":"2024-03-16T12:20:12.790351+0000","last_fullsized":"2024-03-16T12:20:12.790351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:38.070429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.846970+0000","last_change":"2024-03-16T12:18:44.591675+0000","last_active":"2024-03-16T12:19:44.846970+0000","last_peered":"2024-03-16T12:19:44.846970+0000","last_clean":"2024-03-16T12:19:44.846970+0000","last_became_active":"2024-03-16T12:18:44.591258+0000","last_became_peered":"2024-03-16T12:18:44.591258+0000","last_unstale":"2024-03-16T12:19:44.846970+0000","last_undegraded":"2024-03-16T12:19:44.846970+0000","last_fullsized":"2024-03-16T12:19:44.846970+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:11:03.525203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":182,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:32.719508+0000","last_change":"2024-03-16T12:18:39.979118+0000","last_active":"2024-03-16T12:21:32.719508+0000","last_peered":"2024-03-16T12:21:32.719508+0000","last_clean":"2024-03-16T12:21:32.719508+0000","last_became_active":"2024-03-16T12:18:39.978743+0000","last_became_peered":"2024-03-16T12:18:39.978743+0000","last_unstale":"2024-03-16T12:21:32.719508+0000","last_undegraded":"2024-03-16T12:21:32.719508+0000","last_fullsized":"2024-03-16T12:21:32.719508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:16:52.228665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":898,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":151,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.755629+0000","last_change":"2024-03-16T12:18:40.505349+0000","last_active":"2024-03-16T12:20:12.755629+0000","last_peered":"2024-03-16T12:20:12.755629+0000","last_clean":"2024-03-16T12:20:12.755629+0000","last_became_active":"2024-03-16T12:18:40.505028+0000","last_became_peered":"2024-03-16T12:18:40.505028+0000","last_unstale":"2024-03-16T12:20:12.755629+0000","last_undegraded":"2024-03-16T12:20:12.755629+0000","last_fullsized":"2024-03-16T12:20:12.755629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:35:43.800013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567169+0000","last_change":"2024-03-16T12:19:44.565998+0000","last_active":"2024-03-16T12:19:44.567169+0000","last_peered":"2024-03-16T12:19:44.567169+0000","last_clean":"2024-03-16T12:19:44.567169+0000","last_became_active":"2024-03-16T12:19:44.564591+0000","last_became_peered":"2024-03-16T12:19:44.564591+0000","last_unstale":"2024-03-16T12:19:44.567169+0000","last_undegraded":"2024-03-16T12:19:44.567169+0000","last_fullsized":"2024-03-16T12:19:44.567169+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:57:48.227439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543368+0000","last_change":"2024-03-16T12:18:39.992971+0000","last_active":"2024-03-16T12:19:44.543368+0000","last_peered":"2024-03-16T12:19:44.543368+0000","last_clean":"2024-03-16T12:19:44.543368+0000","last_became_active":"2024-03-16T12:18:39.992630+0000","last_became_peered":"2024-03-16T12:18:39.992630+0000","last_unstale":"2024-03-16T12:19:44.543368+0000","last_undegraded":"2024-03-16T12:19:44.543368+0000","last_fullsized":"2024-03-16T12:19:44.543368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:15.935759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.782622+0000","last_change":"2024-03-16T12:18:40.505715+0000","last_active":"2024-03-16T12:20:12.782622+0000","last_peered":"2024-03-16T12:20:12.782622+0000","last_clean":"2024-03-16T12:20:12.782622+0000","last_became_active":"2024-03-16T12:18:40.505397+0000","last_became_peered":"2024-03-16T12:18:40.505397+0000","last_unstale":"2024-03-16T12:20:12.782622+0000","last_undegraded":"2024-03-16T12:20:12.782622+0000","last_fullsized":"2024-03-16T12:20:12.782622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:55.018991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548932+0000","last_change":"2024-03-16T12:18:44.582766+0000","last_active":"2024-03-16T12:19:44.548932+0000","last_peered":"2024-03-16T12:19:44.548932+0000","last_clean":"2024-03-16T12:19:44.548932+0000","last_became_active":"2024-03-16T12:18:44.581069+0000","last_became_peered":"2024-03-16T12:18:44.581069+0000","last_unstale":"2024-03-16T12:19:44.548932+0000","last_undegraded":"2024-03-16T12:19:44.548932+0000","last_fullsized":"2024-03-16T12:19:44.548932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:24:13.613823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543880+0000","last_change":"2024-03-16T12:18:39.979189+0000","last_active":"2024-03-16T12:19:44.543880+0000","last_peered":"2024-03-16T12:19:44.543880+0000","last_clean":"2024-03-16T12:19:44.543880+0000","last_became_active":"2024-03-16T12:18:39.978878+0000","last_became_peered":"2024-03-16T12:18:39.978878+0000","last_unstale":"2024-03-16T12:19:44.543880+0000","last_undegraded":"2024-03-16T12:19:44.543880+0000","last_fullsized":"2024-03-16T12:19:44.543880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:41:53.261863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770720+0000","last_change":"2024-03-16T12:18:40.503697+0000","last_active":"2024-03-16T12:20:12.770720+0000","last_peered":"2024-03-16T12:20:12.770720+0000","last_clean":"2024-03-16T12:20:12.770720+0000","last_became_active":"2024-03-16T12:18:40.503363+0000","last_became_peered":"2024-03-16T12:18:40.503363+0000","last_unstale":"2024-03-16T12:20:12.770720+0000","last_undegraded":"2024-03-16T12:20:12.770720+0000","last_fullsized":"2024-03-16T12:20:12.770720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:10:11.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543490+0000","last_change":"2024-03-16T12:18:44.582383+0000","last_active":"2024-03-16T12:19:44.543490+0000","last_peered":"2024-03-16T12:19:44.543490+0000","last_clean":"2024-03-16T12:19:44.543490+0000","last_became_active":"2024-03-16T12:18:44.581994+0000","last_became_peered":"2024-03-16T12:18:44.581994+0000","last_unstale":"2024-03-16T12:19:44.543490+0000","last_undegraded":"2024-03-16T12:19:44.543490+0000","last_fullsized":"2024-03-16T12:19:44.543490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:38.708773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847452+0000","last_change":"2024-03-16T12:18:39.997313+0000","last_active":"2024-03-16T12:19:44.847452+0000","last_peered":"2024-03-16T12:19:44.847452+0000","last_clean":"2024-03-16T12:19:44.847452+0000","last_became_active":"2024-03-16T12:18:39.996934+0000","last_became_peered":"2024-03-16T12:18:39.996934+0000","last_unstale":"2024-03-16T12:19:44.847452+0000","last_undegraded":"2024-03-16T12:19:44.847452+0000","last_fullsized":"2024-03-16T12:19:44.847452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:55:50.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.793183+0000","last_change":"2024-03-16T12:18:40.520292+0000","last_active":"2024-03-16T12:20:12.793183+0000","last_peered":"2024-03-16T12:20:12.793183+0000","last_clean":"2024-03-16T12:20:12.793183+0000","last_became_active":"2024-03-16T12:18:40.519995+0000","last_became_peered":"2024-03-16T12:18:40.519995+0000","last_unstale":"2024-03-16T12:20:12.793183+0000","last_undegraded":"2024-03-16T12:20:12.793183+0000","last_fullsized":"2024-03-16T12:20:12.793183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:35.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559770+0000","last_change":"2024-03-16T12:18:44.574109+0000","last_active":"2024-03-16T12:19:44.559770+0000","last_peered":"2024-03-16T12:19:44.559770+0000","last_clean":"2024-03-16T12:19:44.559770+0000","last_became_active":"2024-03-16T12:18:44.573597+0000","last_became_peered":"2024-03-16T12:18:44.573597+0000","last_unstale":"2024-03-16T12:19:44.559770+0000","last_undegraded":"2024-03-16T12:19:44.559770+0000","last_fullsized":"2024-03-16T12:19:44.559770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:24:32.957641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547822+0000","last_change":"2024-03-16T12:18:39.987813+0000","last_active":"2024-03-16T12:19:44.547822+0000","last_peered":"2024-03-16T12:19:44.547822+0000","last_clean":"2024-03-16T12:19:44.547822+0000","last_became_active":"2024-03-16T12:18:39.987329+0000","last_became_peered":"2024-03-16T12:18:39.987329+0000","last_unstale":"2024-03-16T12:19:44.547822+0000","last_undegraded":"2024-03-16T12:19:44.547822+0000","last_fullsized":"2024-03-16T12:19:44.547822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:05.984547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786284+0000","last_change":"2024-03-16T12:18:40.504066+0000","last_active":"2024-03-16T12:20:12.786284+0000","last_peered":"2024-03-16T12:20:12.786284+0000","last_clean":"2024-03-16T12:20:12.786284+0000","last_became_active":"2024-03-16T12:18:40.503734+0000","last_became_peered":"2024-03-16T12:18:40.503734+0000","last_unstale":"2024-03-16T12:20:12.786284+0000","last_undegraded":"2024-03-16T12:20:12.786284+0000","last_fullsized":"2024-03-16T12:20:12.786284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:22.384708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559484+0000","last_change":"2024-03-16T12:18:44.576507+0000","last_active":"2024-03-16T12:19:44.559484+0000","last_peered":"2024-03-16T12:19:44.559484+0000","last_clean":"2024-03-16T12:19:44.559484+0000","last_became_active":"2024-03-16T12:18:44.576023+0000","last_became_peered":"2024-03-16T12:18:44.576023+0000","last_unstale":"2024-03-16T12:19:44.559484+0000","last_undegraded":"2024-03-16T12:19:44.559484+0000","last_fullsized":"2024-03-16T12:19:44.559484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:11:28.298573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776444+0000","last_change":"2024-03-16T12:19:44.552241+0000","last_active":"2024-03-16T12:20:12.776444+0000","last_peered":"2024-03-16T12:20:12.776444+0000","last_clean":"2024-03-16T12:20:12.776444+0000","last_became_active":"2024-03-16T12:19:44.551762+0000","last_became_peered":"2024-03-16T12:19:44.551762+0000","last_unstale":"2024-03-16T12:20:12.776444+0000","last_undegraded":"2024-03-16T12:20:12.776444+0000","last_fullsized":"2024-03-16T12:20:12.776444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:24.454587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:21:37.727945+0000","last_change":"2024-03-16T12:18:40.003325+0000","last_active":"2024-03-16T12:21:37.727945+0000","last_peered":"2024-03-16T12:21:37.727945+0000","last_clean":"2024-03-16T12:21:37.727945+0000","last_became_active":"2024-03-16T12:18:40.002974+0000","last_became_peered":"2024-03-16T12:18:40.002974+0000","last_unstale":"2024-03-16T12:21:37.727945+0000","last_undegraded":"2024-03-16T12:21:37.727945+0000","last_fullsized":"2024-03-16T12:21:37.727945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:16.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547991+0000","last_change":"2024-03-16T12:18:44.587677+0000","last_active":"2024-03-16T12:19:44.547991+0000","last_peered":"2024-03-16T12:19:44.547991+0000","last_clean":"2024-03-16T12:19:44.547991+0000","last_became_active":"2024-03-16T12:18:44.587233+0000","last_became_peered":"2024-03-16T12:18:44.587233+0000","last_unstale":"2024-03-16T12:19:44.547991+0000","last_undegraded":"2024-03-16T12:19:44.547991+0000","last_fullsized":"2024-03-16T12:19:44.547991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:42:47.785191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.790357+0000","last_change":"2024-03-16T12:18:40.525038+0000","last_active":"2024-03-16T12:20:12.790357+0000","last_peered":"2024-03-16T12:20:12.790357+0000","last_clean":"2024-03-16T12:20:12.790357+0000","last_became_active":"2024-03-16T12:18:40.521131+0000","last_became_peered":"2024-03-16T12:18:40.521131+0000","last_unstale":"2024-03-16T12:20:12.790357+0000","last_undegraded":"2024-03-16T12:20:12.790357+0000","last_fullsized":"2024-03-16T12:20:12.790357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:14.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546518+0000","last_change":"2024-03-16T12:18:39.989705+0000","last_active":"2024-03-16T12:19:44.546518+0000","last_peered":"2024-03-16T12:19:44.546518+0000","last_clean":"2024-03-16T12:19:44.546518+0000","last_became_active":"2024-03-16T12:18:39.989325+0000","last_became_peered":"2024-03-16T12:18:39.989325+0000","last_unstale":"2024-03-16T12:19:44.546518+0000","last_undegraded":"2024-03-16T12:19:44.546518+0000","last_fullsized":"2024-03-16T12:19:44.546518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:01:41.914081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548608+0000","last_change":"2024-03-16T12:17:20.459287+0000","last_active":"2024-03-16T12:19:44.548608+0000","last_peered":"2024-03-16T12:19:44.548608+0000","last_clean":"2024-03-16T12:19:44.548608+0000","last_became_active":"2024-03-16T12:17:20.458866+0000","last_became_peered":"2024-03-16T12:17:20.458866+0000","last_unstale":"2024-03-16T12:19:44.548608+0000","last_undegraded":"2024-03-16T12:19:44.548608+0000","last_fullsized":"2024-03-16T12:19:44.548608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:13:27.315907+0000","last_clean_scrub_stamp":"2024-03-16T12:13:27.315907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:54.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.567444+0000","last_change":"2024-03-16T12:19:44.566987+0000","last_active":"2024-03-16T12:19:44.567444+0000","last_peered":"2024-03-16T12:19:44.567444+0000","last_clean":"2024-03-16T12:19:44.567444+0000","last_became_active":"2024-03-16T12:19:44.566529+0000","last_became_peered":"2024-03-16T12:19:44.566529+0000","last_unstale":"2024-03-16T12:19:44.567444+0000","last_undegraded":"2024-03-16T12:19:44.567444+0000","last_fullsized":"2024-03-16T12:19:44.567444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:53:03.124434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.785697+0000","last_change":"2024-03-16T12:18:40.507278+0000","last_active":"2024-03-16T12:20:12.785697+0000","last_peered":"2024-03-16T12:20:12.785697+0000","last_clean":"2024-03-16T12:20:12.785697+0000","last_became_active":"2024-03-16T12:18:40.505001+0000","last_became_peered":"2024-03-16T12:18:40.505001+0000","last_unstale":"2024-03-16T12:20:12.785697+0000","last_undegraded":"2024-03-16T12:20:12.785697+0000","last_fullsized":"2024-03-16T12:20:12.785697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:40.426796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548707+0000","last_change":"2024-03-16T12:18:39.979630+0000","last_active":"2024-03-16T12:19:44.548707+0000","last_peered":"2024-03-16T12:19:44.548707+0000","last_clean":"2024-03-16T12:19:44.548707+0000","last_became_active":"2024-03-16T12:18:39.979322+0000","last_became_peered":"2024-03-16T12:18:39.979322+0000","last_unstale":"2024-03-16T12:19:44.548707+0000","last_undegraded":"2024-03-16T12:19:44.548707+0000","last_fullsized":"2024-03-16T12:19:44.548707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:08.505134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559891+0000","last_change":"2024-03-16T12:19:44.559544+0000","last_active":"2024-03-16T12:19:44.559891+0000","last_peered":"2024-03-16T12:19:44.559891+0000","last_clean":"2024-03-16T12:19:44.559891+0000","last_became_active":"2024-03-16T12:19:44.559030+0000","last_became_peered":"2024-03-16T12:19:44.559030+0000","last_unstale":"2024-03-16T12:19:44.559891+0000","last_undegraded":"2024-03-16T12:19:44.559891+0000","last_fullsized":"2024-03-16T12:19:44.559891+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:35:46.496512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7,6],"acting":[11,10,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.543822+0000","last_change":"2024-03-16T12:18:39.992974+0000","last_active":"2024-03-16T12:19:44.543822+0000","last_peered":"2024-03-16T12:19:44.543822+0000","last_clean":"2024-03-16T12:19:44.543822+0000","last_became_active":"2024-03-16T12:18:39.992648+0000","last_became_peered":"2024-03-16T12:18:39.992648+0000","last_unstale":"2024-03-16T12:19:44.543822+0000","last_undegraded":"2024-03-16T12:19:44.543822+0000","last_fullsized":"2024-03-16T12:19:44.543822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:13.203048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.787665+0000","last_change":"2024-03-16T12:18:40.514691+0000","last_active":"2024-03-16T12:20:12.787665+0000","last_peered":"2024-03-16T12:20:12.787665+0000","last_clean":"2024-03-16T12:20:12.787665+0000","last_became_active":"2024-03-16T12:18:40.514358+0000","last_became_peered":"2024-03-16T12:18:40.514358+0000","last_unstale":"2024-03-16T12:20:12.787665+0000","last_undegraded":"2024-03-16T12:20:12.787665+0000","last_fullsized":"2024-03-16T12:20:12.787665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:09.129544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546514+0000","last_change":"2024-03-16T12:18:44.584946+0000","last_active":"2024-03-16T12:19:44.546514+0000","last_peered":"2024-03-16T12:19:44.546514+0000","last_clean":"2024-03-16T12:19:44.546514+0000","last_became_active":"2024-03-16T12:18:44.584441+0000","last_became_peered":"2024-03-16T12:18:44.584441+0000","last_unstale":"2024-03-16T12:19:44.546514+0000","last_undegraded":"2024-03-16T12:19:44.546514+0000","last_fullsized":"2024-03-16T12:19:44.546514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:00:20.878809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:50.091668+0000","last_change":"2024-03-16T12:18:39.990483+0000","last_active":"2024-03-16T12:20:50.091668+0000","last_peered":"2024-03-16T12:20:50.091668+0000","last_clean":"2024-03-16T12:20:50.091668+0000","last_became_active":"2024-03-16T12:18:39.990098+0000","last_became_peered":"2024-03-16T12:18:39.990098+0000","last_unstale":"2024-03-16T12:20:50.091668+0000","last_undegraded":"2024-03-16T12:20:50.091668+0000","last_fullsized":"2024-03-16T12:20:50.091668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:35:10.543853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.788697+0000","last_change":"2024-03-16T12:18:40.522924+0000","last_active":"2024-03-16T12:20:12.788697+0000","last_peered":"2024-03-16T12:20:12.788697+0000","last_clean":"2024-03-16T12:20:12.788697+0000","last_became_active":"2024-03-16T12:18:40.521156+0000","last_became_peered":"2024-03-16T12:18:40.521156+0000","last_unstale":"2024-03-16T12:20:12.788697+0000","last_undegraded":"2024-03-16T12:20:12.788697+0000","last_fullsized":"2024-03-16T12:20:12.788697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:58:34.824590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546828+0000","last_change":"2024-03-16T12:18:44.569660+0000","last_active":"2024-03-16T12:19:44.546828+0000","last_peered":"2024-03-16T12:19:44.546828+0000","last_clean":"2024-03-16T12:19:44.546828+0000","last_became_active":"2024-03-16T12:18:44.569185+0000","last_became_peered":"2024-03-16T12:18:44.569185+0000","last_unstale":"2024-03-16T12:19:44.546828+0000","last_undegraded":"2024-03-16T12:19:44.546828+0000","last_fullsized":"2024-03-16T12:19:44.546828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:18.341827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546979+0000","last_change":"2024-03-16T12:18:39.987884+0000","last_active":"2024-03-16T12:19:44.546979+0000","last_peered":"2024-03-16T12:19:44.546979+0000","last_clean":"2024-03-16T12:19:44.546979+0000","last_became_active":"2024-03-16T12:18:39.987427+0000","last_became_peered":"2024-03-16T12:18:39.987427+0000","last_unstale":"2024-03-16T12:19:44.546979+0000","last_undegraded":"2024-03-16T12:19:44.546979+0000","last_fullsized":"2024-03-16T12:19:44.546979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:13.502105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804936+0000","last_change":"2024-03-16T12:18:40.526077+0000","last_active":"2024-03-16T12:20:12.804936+0000","last_peered":"2024-03-16T12:20:12.804936+0000","last_clean":"2024-03-16T12:20:12.804936+0000","last_became_active":"2024-03-16T12:18:40.525550+0000","last_became_peered":"2024-03-16T12:18:40.525550+0000","last_unstale":"2024-03-16T12:20:12.804936+0000","last_undegraded":"2024-03-16T12:20:12.804936+0000","last_fullsized":"2024-03-16T12:20:12.804936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:00.485858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546486+0000","last_change":"2024-03-16T12:18:44.590283+0000","last_active":"2024-03-16T12:19:44.546486+0000","last_peered":"2024-03-16T12:19:44.546486+0000","last_clean":"2024-03-16T12:19:44.546486+0000","last_became_active":"2024-03-16T12:18:44.589777+0000","last_became_peered":"2024-03-16T12:18:44.589777+0000","last_unstale":"2024-03-16T12:19:44.546486+0000","last_undegraded":"2024-03-16T12:19:44.546486+0000","last_fullsized":"2024-03-16T12:19:44.546486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:23:13.285801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168879+0000","last_change":"2024-03-16T12:18:39.989938+0000","last_active":"2024-03-16T12:19:45.168879+0000","last_peered":"2024-03-16T12:19:45.168879+0000","last_clean":"2024-03-16T12:19:45.168879+0000","last_became_active":"2024-03-16T12:18:39.989562+0000","last_became_peered":"2024-03-16T12:18:39.989562+0000","last_unstale":"2024-03-16T12:19:45.168879+0000","last_undegraded":"2024-03-16T12:19:45.168879+0000","last_fullsized":"2024-03-16T12:19:45.168879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:34.225891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.748250+0000","last_change":"2024-03-16T12:18:40.498763+0000","last_active":"2024-03-16T12:20:12.748250+0000","last_peered":"2024-03-16T12:20:12.748250+0000","last_clean":"2024-03-16T12:20:12.748250+0000","last_became_active":"2024-03-16T12:18:40.498397+0000","last_became_peered":"2024-03-16T12:18:40.498397+0000","last_unstale":"2024-03-16T12:20:12.748250+0000","last_undegraded":"2024-03-16T12:20:12.748250+0000","last_fullsized":"2024-03-16T12:20:12.748250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:57.418953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.847167+0000","last_change":"2024-03-16T12:18:44.589035+0000","last_active":"2024-03-16T12:19:44.847167+0000","last_peered":"2024-03-16T12:19:44.847167+0000","last_clean":"2024-03-16T12:19:44.847167+0000","last_became_active":"2024-03-16T12:18:44.588608+0000","last_became_peered":"2024-03-16T12:18:44.588608+0000","last_unstale":"2024-03-16T12:19:44.847167+0000","last_undegraded":"2024-03-16T12:19:44.847167+0000","last_fullsized":"2024-03-16T12:19:44.847167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:40:22.418446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546586+0000","last_change":"2024-03-16T12:18:39.993442+0000","last_active":"2024-03-16T12:19:44.546586+0000","last_peered":"2024-03-16T12:19:44.546586+0000","last_clean":"2024-03-16T12:19:44.546586+0000","last_became_active":"2024-03-16T12:18:39.992396+0000","last_became_peered":"2024-03-16T12:18:39.992396+0000","last_unstale":"2024-03-16T12:19:44.546586+0000","last_undegraded":"2024-03-16T12:19:44.546586+0000","last_fullsized":"2024-03-16T12:19:44.546586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:05.457661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.803505+0000","last_change":"2024-03-16T12:18:40.517961+0000","last_active":"2024-03-16T12:20:12.803505+0000","last_peered":"2024-03-16T12:20:12.803505+0000","last_clean":"2024-03-16T12:20:12.803505+0000","last_became_active":"2024-03-16T12:18:40.517485+0000","last_became_peered":"2024-03-16T12:18:40.517485+0000","last_unstale":"2024-03-16T12:20:12.803505+0000","last_undegraded":"2024-03-16T12:20:12.803505+0000","last_fullsized":"2024-03-16T12:20:12.803505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:41:06.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546670+0000","last_change":"2024-03-16T12:18:44.590252+0000","last_active":"2024-03-16T12:19:44.546670+0000","last_peered":"2024-03-16T12:19:44.546670+0000","last_clean":"2024-03-16T12:19:44.546670+0000","last_became_active":"2024-03-16T12:18:44.589840+0000","last_became_peered":"2024-03-16T12:18:44.589840+0000","last_unstale":"2024-03-16T12:19:44.546670+0000","last_undegraded":"2024-03-16T12:19:44.546670+0000","last_fullsized":"2024-03-16T12:19:44.546670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:19.066714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.547065+0000","last_change":"2024-03-16T12:18:39.978194+0000","last_active":"2024-03-16T12:19:44.547065+0000","last_peered":"2024-03-16T12:19:44.547065+0000","last_clean":"2024-03-16T12:19:44.547065+0000","last_became_active":"2024-03-16T12:18:39.977836+0000","last_became_peered":"2024-03-16T12:18:39.977836+0000","last_unstale":"2024-03-16T12:19:44.547065+0000","last_undegraded":"2024-03-16T12:19:44.547065+0000","last_fullsized":"2024-03-16T12:19:44.547065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:42:11.580323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805338+0000","last_change":"2024-03-16T12:18:40.526700+0000","last_active":"2024-03-16T12:20:12.805338+0000","last_peered":"2024-03-16T12:20:12.805338+0000","last_clean":"2024-03-16T12:20:12.805338+0000","last_became_active":"2024-03-16T12:18:40.525612+0000","last_became_peered":"2024-03-16T12:18:40.525612+0000","last_unstale":"2024-03-16T12:20:12.805338+0000","last_undegraded":"2024-03-16T12:20:12.805338+0000","last_fullsized":"2024-03-16T12:20:12.805338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:46:08.998260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.546343+0000","last_change":"2024-03-16T12:18:44.579183+0000","last_active":"2024-03-16T12:19:44.546343+0000","last_peered":"2024-03-16T12:19:44.546343+0000","last_clean":"2024-03-16T12:19:44.546343+0000","last_became_active":"2024-03-16T12:18:44.578752+0000","last_became_peered":"2024-03-16T12:18:44.578752+0000","last_unstale":"2024-03-16T12:19:44.546343+0000","last_undegraded":"2024-03-16T12:19:44.546343+0000","last_fullsized":"2024-03-16T12:19:44.546343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:09:07.147191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.555974+0000","last_change":"2024-03-16T12:19:44.555974+0000","last_active":"2024-03-16T12:19:44.555974+0000","last_peered":"2024-03-16T12:19:44.555974+0000","last_clean":"2024-03-16T12:19:44.555974+0000","last_became_active":"2024-03-16T12:19:44.550254+0000","last_became_peered":"2024-03-16T12:19:44.550254+0000","last_unstale":"2024-03-16T12:19:44.555974+0000","last_undegraded":"2024-03-16T12:19:44.555974+0000","last_fullsized":"2024-03-16T12:19:44.555974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:06:14.216787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.789538+0000","last_change":"2024-03-16T12:18:40.508195+0000","last_active":"2024-03-16T12:20:12.789538+0000","last_peered":"2024-03-16T12:20:12.789538+0000","last_clean":"2024-03-16T12:20:12.789538+0000","last_became_active":"2024-03-16T12:18:40.507817+0000","last_became_peered":"2024-03-16T12:18:40.507817+0000","last_unstale":"2024-03-16T12:20:12.789538+0000","last_undegraded":"2024-03-16T12:20:12.789538+0000","last_fullsized":"2024-03-16T12:20:12.789538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:39.651695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.168880+0000","last_change":"2024-03-16T12:18:44.579826+0000","last_active":"2024-03-16T12:19:45.168880+0000","last_peered":"2024-03-16T12:19:45.168880+0000","last_clean":"2024-03-16T12:19:45.168880+0000","last_became_active":"2024-03-16T12:18:44.579374+0000","last_became_peered":"2024-03-16T12:18:44.579374+0000","last_unstale":"2024-03-16T12:19:45.168880+0000","last_undegraded":"2024-03-16T12:19:45.168880+0000","last_fullsized":"2024-03-16T12:19:45.168880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:26:50.333801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.548184+0000","last_change":"2024-03-16T12:18:39.989735+0000","last_active":"2024-03-16T12:19:44.548184+0000","last_peered":"2024-03-16T12:19:44.548184+0000","last_clean":"2024-03-16T12:19:44.548184+0000","last_became_active":"2024-03-16T12:18:39.989425+0000","last_became_peered":"2024-03-16T12:18:39.989425+0000","last_unstale":"2024-03-16T12:19:44.548184+0000","last_undegraded":"2024-03-16T12:19:44.548184+0000","last_fullsized":"2024-03-16T12:19:44.548184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:16.179770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.786719+0000","last_change":"2024-03-16T12:18:40.502515+0000","last_active":"2024-03-16T12:20:12.786719+0000","last_peered":"2024-03-16T12:20:12.786719+0000","last_clean":"2024-03-16T12:20:12.786719+0000","last_became_active":"2024-03-16T12:18:40.502166+0000","last_became_peered":"2024-03-16T12:18:40.502166+0000","last_unstale":"2024-03-16T12:20:12.786719+0000","last_undegraded":"2024-03-16T12:20:12.786719+0000","last_fullsized":"2024-03-16T12:20:12.786719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:30:34.581316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:45.172049+0000","last_change":"2024-03-16T12:18:44.591171+0000","last_active":"2024-03-16T12:19:45.172049+0000","last_peered":"2024-03-16T12:19:45.172049+0000","last_clean":"2024-03-16T12:19:45.172049+0000","last_became_active":"2024-03-16T12:18:44.590072+0000","last_became_peered":"2024-03-16T12:18:44.590072+0000","last_unstale":"2024-03-16T12:19:45.172049+0000","last_undegraded":"2024-03-16T12:19:45.172049+0000","last_fullsized":"2024-03-16T12:19:45.172049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:20.816538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.550000+0000","last_change":"2024-03-16T12:18:39.988456+0000","last_active":"2024-03-16T12:19:44.550000+0000","last_peered":"2024-03-16T12:19:44.550000+0000","last_clean":"2024-03-16T12:19:44.550000+0000","last_became_active":"2024-03-16T12:18:39.988125+0000","last_became_peered":"2024-03-16T12:18:39.988125+0000","last_unstale":"2024-03-16T12:19:44.550000+0000","last_undegraded":"2024-03-16T12:19:44.550000+0000","last_fullsized":"2024-03-16T12:19:44.550000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:48.637596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.791712+0000","last_change":"2024-03-16T12:18:40.520393+0000","last_active":"2024-03-16T12:20:12.791712+0000","last_peered":"2024-03-16T12:20:12.791712+0000","last_clean":"2024-03-16T12:20:12.791712+0000","last_became_active":"2024-03-16T12:18:40.519985+0000","last_became_peered":"2024-03-16T12:18:40.519985+0000","last_unstale":"2024-03-16T12:20:12.791712+0000","last_undegraded":"2024-03-16T12:20:12.791712+0000","last_fullsized":"2024-03-16T12:20:12.791712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:40:52.442070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.549274+0000","last_change":"2024-03-16T12:18:44.567969+0000","last_active":"2024-03-16T12:19:44.549274+0000","last_peered":"2024-03-16T12:19:44.549274+0000","last_clean":"2024-03-16T12:19:44.549274+0000","last_became_active":"2024-03-16T12:18:44.566987+0000","last_became_peered":"2024-03-16T12:18:44.566987+0000","last_unstale":"2024-03-16T12:19:44.549274+0000","last_undegraded":"2024-03-16T12:19:44.549274+0000","last_fullsized":"2024-03-16T12:19:44.549274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:43.535202+0000","last_clean_scrub_stamp":"2024-03-16T12:18:43.535202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:51:41.444027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:19:44.559640+0000","last_change":"2024-03-16T12:18:39.998577+0000","last_active":"2024-03-16T12:19:44.559640+0000","last_peered":"2024-03-16T12:19:44.559640+0000","last_clean":"2024-03-16T12:19:44.559640+0000","last_became_active":"2024-03-16T12:18:39.998200+0000","last_became_peered":"2024-03-16T12:18:39.998200+0000","last_unstale":"2024-03-16T12:19:44.559640+0000","last_undegraded":"2024-03-16T12:19:44.559640+0000","last_fullsized":"2024-03-16T12:19:44.559640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:38.919720+0000","last_clean_scrub_stamp":"2024-03-16T12:18:38.919720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:45:26.573708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.771744+0000","last_change":"2024-03-16T12:18:40.505883+0000","last_active":"2024-03-16T12:20:12.771744+0000","last_peered":"2024-03-16T12:20:12.771744+0000","last_clean":"2024-03-16T12:20:12.771744+0000","last_became_active":"2024-03-16T12:18:40.504789+0000","last_became_peered":"2024-03-16T12:18:40.504789+0000","last_unstale":"2024-03-16T12:20:12.771744+0000","last_undegraded":"2024-03-16T12:20:12.771744+0000","last_fullsized":"2024-03-16T12:20:12.771744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:29:25.894575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.774255+0000","last_change":"2024-03-16T12:18:40.505052+0000","last_active":"2024-03-16T12:20:12.774255+0000","last_peered":"2024-03-16T12:20:12.774255+0000","last_clean":"2024-03-16T12:20:12.774255+0000","last_became_active":"2024-03-16T12:18:40.504691+0000","last_became_peered":"2024-03-16T12:18:40.504691+0000","last_unstale":"2024-03-16T12:20:12.774255+0000","last_undegraded":"2024-03-16T12:20:12.774255+0000","last_fullsized":"2024-03-16T12:20:12.774255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:05:50.415954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781112+0000","last_change":"2024-03-16T12:19:44.847899+0000","last_active":"2024-03-16T12:20:12.781112+0000","last_peered":"2024-03-16T12:20:12.781112+0000","last_clean":"2024-03-16T12:20:12.781112+0000","last_became_active":"2024-03-16T12:19:44.847589+0000","last_became_peered":"2024-03-16T12:19:44.847589+0000","last_unstale":"2024-03-16T12:20:12.781112+0000","last_undegraded":"2024-03-16T12:20:12.781112+0000","last_fullsized":"2024-03-16T12:20:12.781112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:59.091516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.802828+0000","last_change":"2024-03-16T12:18:40.512872+0000","last_active":"2024-03-16T12:20:12.802828+0000","last_peered":"2024-03-16T12:20:12.802828+0000","last_clean":"2024-03-16T12:20:12.802828+0000","last_became_active":"2024-03-16T12:18:40.512505+0000","last_became_peered":"2024-03-16T12:18:40.512505+0000","last_unstale":"2024-03-16T12:20:12.802828+0000","last_undegraded":"2024-03-16T12:20:12.802828+0000","last_fullsized":"2024-03-16T12:20:12.802828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:59.054039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.804518+0000","last_change":"2024-03-16T12:18:40.516559+0000","last_active":"2024-03-16T12:20:12.804518+0000","last_peered":"2024-03-16T12:20:12.804518+0000","last_clean":"2024-03-16T12:20:12.804518+0000","last_became_active":"2024-03-16T12:18:40.516098+0000","last_became_peered":"2024-03-16T12:18:40.516098+0000","last_unstale":"2024-03-16T12:20:12.804518+0000","last_undegraded":"2024-03-16T12:20:12.804518+0000","last_fullsized":"2024-03-16T12:20:12.804518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:38.586537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.805439+0000","last_change":"2024-03-16T12:18:40.506992+0000","last_active":"2024-03-16T12:20:12.805439+0000","last_peered":"2024-03-16T12:20:12.805439+0000","last_clean":"2024-03-16T12:20:12.805439+0000","last_became_active":"2024-03-16T12:18:40.506634+0000","last_became_peered":"2024-03-16T12:18:40.506634+0000","last_unstale":"2024-03-16T12:20:12.805439+0000","last_undegraded":"2024-03-16T12:20:12.805439+0000","last_fullsized":"2024-03-16T12:20:12.805439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:39.478512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.770556+0000","last_change":"2024-03-16T12:18:40.502895+0000","last_active":"2024-03-16T12:20:12.770556+0000","last_peered":"2024-03-16T12:20:12.770556+0000","last_clean":"2024-03-16T12:20:12.770556+0000","last_became_active":"2024-03-16T12:18:40.502575+0000","last_became_peered":"2024-03-16T12:18:40.502575+0000","last_unstale":"2024-03-16T12:20:12.770556+0000","last_undegraded":"2024-03-16T12:20:12.770556+0000","last_fullsized":"2024-03-16T12:20:12.770556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:21:00.706573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.776877+0000","last_change":"2024-03-16T12:18:40.502527+0000","last_active":"2024-03-16T12:20:12.776877+0000","last_peered":"2024-03-16T12:20:12.776877+0000","last_clean":"2024-03-16T12:20:12.776877+0000","last_became_active":"2024-03-16T12:18:40.502197+0000","last_became_peered":"2024-03-16T12:18:40.502197+0000","last_unstale":"2024-03-16T12:20:12.776877+0000","last_undegraded":"2024-03-16T12:20:12.776877+0000","last_fullsized":"2024-03-16T12:20:12.776877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:39.798662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.777318+0000","last_change":"2024-03-16T12:18:40.522256+0000","last_active":"2024-03-16T12:20:12.777318+0000","last_peered":"2024-03-16T12:20:12.777318+0000","last_clean":"2024-03-16T12:20:12.777318+0000","last_became_active":"2024-03-16T12:18:40.521867+0000","last_became_peered":"2024-03-16T12:18:40.521867+0000","last_unstale":"2024-03-16T12:20:12.777318+0000","last_undegraded":"2024-03-16T12:20:12.777318+0000","last_fullsized":"2024-03-16T12:20:12.777318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:07.463359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.799957+0000","last_change":"2024-03-16T12:18:40.509714+0000","last_active":"2024-03-16T12:20:12.799957+0000","last_peered":"2024-03-16T12:20:12.799957+0000","last_clean":"2024-03-16T12:20:12.799957+0000","last_became_active":"2024-03-16T12:18:40.509403+0000","last_became_peered":"2024-03-16T12:18:40.509403+0000","last_unstale":"2024-03-16T12:20:12.799957+0000","last_undegraded":"2024-03-16T12:20:12.799957+0000","last_fullsized":"2024-03-16T12:20:12.799957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:50.007448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.760742+0000","last_change":"2024-03-16T12:18:40.523280+0000","last_active":"2024-03-16T12:20:12.760742+0000","last_peered":"2024-03-16T12:20:12.760742+0000","last_clean":"2024-03-16T12:20:12.760742+0000","last_became_active":"2024-03-16T12:18:40.520012+0000","last_became_peered":"2024-03-16T12:18:40.520012+0000","last_unstale":"2024-03-16T12:20:12.760742+0000","last_undegraded":"2024-03-16T12:20:12.760742+0000","last_fullsized":"2024-03-16T12:20:12.760742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:19:22.737379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.800264+0000","last_change":"2024-03-16T12:18:40.513098+0000","last_active":"2024-03-16T12:20:12.800264+0000","last_peered":"2024-03-16T12:20:12.800264+0000","last_clean":"2024-03-16T12:20:12.800264+0000","last_became_active":"2024-03-16T12:18:40.512776+0000","last_became_peered":"2024-03-16T12:18:40.512776+0000","last_unstale":"2024-03-16T12:20:12.800264+0000","last_undegraded":"2024-03-16T12:20:12.800264+0000","last_fullsized":"2024-03-16T12:20:12.800264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:09.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.806609+0000","last_change":"2024-03-16T12:18:40.520294+0000","last_active":"2024-03-16T12:20:12.806609+0000","last_peered":"2024-03-16T12:20:12.806609+0000","last_clean":"2024-03-16T12:20:12.806609+0000","last_became_active":"2024-03-16T12:18:40.518188+0000","last_became_peered":"2024-03-16T12:18:40.518188+0000","last_unstale":"2024-03-16T12:20:12.806609+0000","last_undegraded":"2024-03-16T12:20:12.806609+0000","last_fullsized":"2024-03-16T12:20:12.806609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:14:25.819199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-16T12:20:12.781562+0000","last_change":"2024-03-16T12:19:44.849288+0000","last_active":"2024-03-16T12:20:12.781562+0000","last_peered":"2024-03-16T12:20:12.781562+0000","last_clean":"2024-03-16T12:20:12.781562+0000","last_became_active":"2024-03-16T12:19:44.848869+0000","last_became_peered":"2024-03-16T12:19:44.848869+0000","last_unstale":"2024-03-16T12:20:12.781562+0000","last_undegraded":"2024-03-16T12:20:12.781562+0000","last_fullsized":"2024-03-16T12:20:12.781562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T12:18:39.961730+0000","last_clean_scrub_stamp":"2024-03-16T12:18:39.961730+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:36.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72185,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":929,"num_read_kb":541,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":216555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":500849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28696,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716456,"statfs":{"total":95995035648,"available":95965650944,"internally_reserved":0,"allocated":1720320,"data_stored":1156658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":500971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1096799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495693,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":564402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":565718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":501187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36940,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708212,"statfs":{"total":95995035648,"available":95957209088,"internally_reserved":0,"allocated":1642496,"data_stored":1092077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T12:21:40.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 -- 172.21.15.98:0/1093740082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 msgr2=0x7f45d0110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 --2- 172.21.15.98:0/1093740082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d0110060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009be0 tx=0x7f45c802bef0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 -- 172.21.15.98:0/1093740082 shutdown_connections 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 --2- 172.21.15.98:0/1093740082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d0110060 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 --2- 172.21.15.98:0/1093740082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d0109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 --2- 172.21.15.98:0/1093740082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d0108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 -- 172.21.15.98:0/1093740082 >> 172.21.15.98:0/1093740082 conn(0x7f45d0076bb0 msgr2=0x7f45d0078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 -- 172.21.15.98:0/1093740082 shutdown_connections 2024-03-16T12:21:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.626+0000 7f45d5000640 1 -- 172.21.15.98:0/1093740082 wait complete. 2024-03-16T12:21:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.627+0000 7f45d5000640 1 Processor -- start 2024-03-16T12:21:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.627+0000 7f45d5000640 1 -- start start 2024-03-16T12:21:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.627+0000 7f45d5000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45d5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45d5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d01a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45d5000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01138a0 con 0x7f45d0108e10 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45d5000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0113720 con 0x7f45d0107c20 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45d5000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0113a20 con 0x7f45d01097d0 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45c6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54102/0 (socket says 172.21.15.98:54102) 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 -- 172.21.15.98:0/3449775846 learned_addr learned my addr 172.21.15.98:0/3449775846 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cfe00640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d01a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 -- 172.21.15.98:0/3449775846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 msgr2=0x7f45d01a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d01a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 -- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 msgr2=0x7f45d019e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d019e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 -- 172.21.15.98:0/3449775846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d01a5a70 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45c6a00640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d019e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.628+0000 7f45cf400640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e9f0 tx=0x7f45c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.629+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c000ccc0 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.629+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c00035f0 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.629+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c000ce60 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.629+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01a5d60 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.629+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01a61f0 con 0x7f45d0107c20 2024-03-16T12:21:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.630+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d010eee0 con 0x7f45d0107c20 2024-03-16T12:21:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.632+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f45c0003b10 con 0x7f45d0107c20 2024-03-16T12:21:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.633+0000 7f45ccc00640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.634+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f45c00ceff0 con 0x7f45d0107c20 2024-03-16T12:21:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.634+0000 7f45c6a00640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.634+0000 7f45c6a00640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 0x7f45b40a4d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45d019f360 tx=0x7f45b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.635+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45c009bd30 con 0x7f45d0107c20 2024-03-16T12:21:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.837+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45d00624f0 con 0x7f45d0107c20 2024-03-16T12:21:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.840+0000 7f45ccc00640 1 -- 172.21.15.98:0/3449775846 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f45c009b630 con 0x7f45d0107c20 2024-03-16T12:21:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:40.822 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 msgr2=0x7f45b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 0x7f45b40a4d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45d019f360 tx=0x7f45b8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 msgr2=0x7f45d019de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e9f0 tx=0x7f45c000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 shutdown_connections 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d01097d0 0x7f45d01a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d0108e10 0x7f45d019e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 --2- 172.21.15.98:0/3449775846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d0107c20 0x7f45d019de50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:40.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 >> 172.21.15.98:0/3449775846 conn(0x7f45d0076bb0 msgr2=0x7f45d010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 shutdown_connections 2024-03-16T12:21:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:40.843+0000 7f45d5000640 1 -- 172.21.15.98:0/3449775846 wait complete. 2024-03-16T12:21:40.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:21:40.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:21:41.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 -- 172.21.15.98:0/3847966557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 msgr2=0x7faa30101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 --2- 172.21.15.98:0/3847966557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30101c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa20009900 tx=0x7faa2002bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 -- 172.21.15.98:0/3847966557 shutdown_connections 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 --2- 172.21.15.98:0/3847966557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 0x7faa30109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 --2- 172.21.15.98:0/3847966557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa30102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 --2- 172.21.15.98:0/3847966557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30101c10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 -- 172.21.15.98:0/3847966557 >> 172.21.15.98:0/3847966557 conn(0x7faa300fcd60 msgr2=0x7faa300ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 -- 172.21.15.98:0/3847966557 shutdown_connections 2024-03-16T12:21:41.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.082+0000 7faa34a00640 1 -- 172.21.15.98:0/3847966557 wait complete. 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 Processor -- start 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 -- start start 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 0x7faa301a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d490 con 0x7faa301033a0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d310 con 0x7faa301017f0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.083+0000 7faa34a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d610 con 0x7faa301029e0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa2de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56264/0 (socket says 172.21.15.98:56264) 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa2e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 0x7faa301a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 -- 172.21.15.98:0/3498260900 learned_addr learned my addr 172.21.15.98:0/3498260900 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 -- 172.21.15.98:0/3498260900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 msgr2=0x7faa30199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 -- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 msgr2=0x7faa301a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 0x7faa301a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 -- 172.21.15.98:0/3498260900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a1970 con 0x7faa301029e0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.084+0000 7faa25400640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faa1800ea60 tx=0x7faa1800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.085+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1800cde0 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.085+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa180035f0 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.085+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18010670 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.085+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a1c60 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.085+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a21b0 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.086+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa301a23e0 con 0x7faa301029e0 2024-03-16T12:21:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.087+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa180037a0 con 0x7faa301029e0 2024-03-16T12:21:41.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.089+0000 7faa27400640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 0x7faa140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.089+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faa180cf100 con 0x7faa301029e0 2024-03-16T12:21:41.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.089+0000 7faa2de00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 0x7faa140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.090+0000 7faa2de00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 0x7faa140a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa2002c170 tx=0x7faa2002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.093+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa1809bde0 con 0x7faa301029e0 2024-03-16T12:21:41.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:40 smithi098 ceph-mon[31633]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:40 smithi098 ceph-mon[31633]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:40 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3449775846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:41.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:40 smithi153 ceph-mon[36901]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:40 smithi153 ceph-mon[36901]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:40 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3449775846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:41.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.289+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa301a25b0 con 0x7faa301029e0 2024-03-16T12:21:41.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.290+0000 7faa27400640 1 -- 172.21.15.98:0/3498260900 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7faa180860f0 con 0x7faa301029e0 2024-03-16T12:21:41.271 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:21:41.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 msgr2=0x7faa140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 0x7faa140a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa2002c170 tx=0x7faa2002df90 comp rx=0 tx=0).stop 2024-03-16T12:21:41.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 msgr2=0x7faa3019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faa1800ea60 tx=0x7faa1800ef20 comp rx=0 tx=0).stop 2024-03-16T12:21:41.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 shutdown_connections 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faa140a2800 0x7faa140a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa301033a0 0x7faa301a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa301029e0 0x7faa3019a290 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 --2- 172.21.15.98:0/3498260900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa301017f0 0x7faa30199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 >> 172.21.15.98:0/3498260900 conn(0x7faa300fcd60 msgr2=0x7faa30073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 shutdown_connections 2024-03-16T12:21:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.293+0000 7faa34a00640 1 -- 172.21.15.98:0/3498260900 wait complete. 2024-03-16T12:21:41.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:21:41.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:40 smithi173 ceph-mon[36632]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:40 smithi173 ceph-mon[36632]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:40 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3449775846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 -- 172.21.15.98:0/2487983633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8107e00 msgr2=0x7f8fe8108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2487983633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8107e00 0x7f8fe8108220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009860 tx=0x7f8fdc02bb70 comp rx=0 tx=0).stop 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 -- 172.21.15.98:0/2487983633 shutdown_connections 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2487983633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe81099b0 0x7f8fe8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2487983633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe8108ff0 0x7f8fe8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2487983633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8107e00 0x7f8fe8108220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 -- 172.21.15.98:0/2487983633 >> 172.21.15.98:0/2487983633 conn(0x7f8fe8076da0 msgr2=0x7f8fe80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 -- 172.21.15.98:0/2487983633 shutdown_connections 2024-03-16T12:21:41.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.530+0000 7f8fefe00640 1 -- 172.21.15.98:0/2487983633 wait complete. 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.531+0000 7f8fefe00640 1 Processor -- start 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.531+0000 7f8fefe00640 1 -- start start 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 0x7f8fe819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 0x7f8fe81a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe8113a80 con 0x7f8fe81099b0 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe8113900 con 0x7f8fe8108ff0 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fefe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe8113c00 con 0x7f8fe8107e00 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8feca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 0x7f8fe819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56284/0 (socket says 172.21.15.98:56284) 2024-03-16T12:21:41.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 -- 172.21.15.98:0/2352143943 learned_addr learned my addr 172.21.15.98:0/2352143943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fede00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 0x7f8fe81a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 -- 172.21.15.98:0/2352143943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 msgr2=0x7f8fe819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 0x7f8fe819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 -- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 msgr2=0x7f8fe81a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 0x7f8fe81a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fed400640 1 -- 172.21.15.98:0/2352143943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe81a5d20 con 0x7f8fe8107e00 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8fede00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 0x7f8fe81a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.532+0000 7f8feca00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 0x7f8fe819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fed400640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0349e0 tx=0x7f8fdc032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc041020 con 0x7f8fe8107e00 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fdc02c9f0 con 0x7f8fe8107e00 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe81a5fb0 con 0x7f8fe8107e00 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc039040 con 0x7f8fe8107e00 2024-03-16T12:21:41.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.533+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe81a63b0 con 0x7f8fe8107e00 2024-03-16T12:21:41.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.534+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe810f1a0 con 0x7f8fe8107e00 2024-03-16T12:21:41.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.536+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8fdc044020 con 0x7f8fe8107e00 2024-03-16T12:21:41.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.537+0000 7f8fe6000640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 0x7f8fd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.537+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8fdc0f0710 con 0x7f8fe8107e00 2024-03-16T12:21:41.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.537+0000 7f8feca00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 0x7f8fd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.538+0000 7f8feca00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 0x7f8fd00a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fe819f610 tx=0x7f8fd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.541+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fdc04b020 con 0x7f8fe8107e00 2024-03-16T12:21:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.739+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8fe80623e0 con 0x7f8fe8107e00 2024-03-16T12:21:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.740+0000 7f8fe6000640 1 -- 172.21.15.98:0/2352143943 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f8fdc0bd3f0 con 0x7f8fe8107e00 2024-03-16T12:21:41.721 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.742+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 msgr2=0x7f8fd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 0x7f8fd00a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fe819f610 tx=0x7f8fd8005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 msgr2=0x7f8fe819e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0349e0 tx=0x7f8fdc032430 comp rx=0 tx=0).stop 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 shutdown_connections 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8fd00a2800 0x7f8fd00a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe81099b0 0x7f8fe81a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe8108ff0 0x7f8fe819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 --2- 172.21.15.98:0/2352143943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe8107e00 0x7f8fe819e100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 >> 172.21.15.98:0/2352143943 conn(0x7f8fe8076da0 msgr2=0x7f8fe810c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 shutdown_connections 2024-03-16T12:21:41.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.743+0000 7f8fefe00640 1 -- 172.21.15.98:0/2352143943 wait complete. 2024-03-16T12:21:41.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:21:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.970+0000 7f0565400640 1 -- 172.21.15.98:0/2785155466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560108ff0 msgr2=0x7f0560109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.970+0000 7f0565400640 1 --2- 172.21.15.98:0/2785155466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560108ff0 0x7f0560109470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0548009900 tx=0x7f054802bc40 comp rx=0 tx=0).stop 2024-03-16T12:21:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 -- 172.21.15.98:0/2785155466 shutdown_connections 2024-03-16T12:21:41.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 --2- 172.21.15.98:0/2785155466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 0x7f0560110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 --2- 172.21.15.98:0/2785155466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560108ff0 0x7f0560109470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 --2- 172.21.15.98:0/2785155466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560107e00 0x7f0560108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 -- 172.21.15.98:0/2785155466 >> 172.21.15.98:0/2785155466 conn(0x7f0560076da0 msgr2=0x7f05600791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 -- 172.21.15.98:0/2785155466 shutdown_connections 2024-03-16T12:21:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.971+0000 7f0565400640 1 -- 172.21.15.98:0/2785155466 wait complete. 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 Processor -- start 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 -- start start 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 0x7f056019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 0x7f05601a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0560113a80 con 0x7f0560108ff0 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0560113900 con 0x7f05601099b0 2024-03-16T12:21:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0565400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0560113c00 con 0x7f0560107e00 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f055e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f055e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56310/0 (socket says 172.21.15.98:56310) 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f055e800640 1 -- 172.21.15.98:0/402909554 learned_addr learned my addr 172.21.15.98:0/402909554 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f055f200640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 0x7f05601a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.972+0000 7f0557e00640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 0x7f056019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 -- 172.21.15.98:0/402909554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 msgr2=0x7f05601a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 0x7f05601a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 -- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 msgr2=0x7f056019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 0x7f056019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 -- 172.21.15.98:0/402909554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05601a5cf0 con 0x7f0560107e00 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0557e00640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 0x7f056019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f055e800640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f055000e9c0 tx=0x7f055000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055000cde0 con 0x7f0560107e00 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550010910 con 0x7f0560107e00 2024-03-16T12:21:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05601a5fe0 con 0x7f0560107e00 2024-03-16T12:21:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550018a00 con 0x7f0560107e00 2024-03-16T12:21:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.973+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05601a6440 con 0x7f0560107e00 2024-03-16T12:21:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.974+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056010f120 con 0x7f0560107e00 2024-03-16T12:21:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.976+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f055001f020 con 0x7f0560107e00 2024-03-16T12:21:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.977+0000 7f0557400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 0x7f05440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.977+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f05500ceff0 con 0x7f0560107e00 2024-03-16T12:21:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.977+0000 7f0557e00640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 0x7f05440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.978+0000 7f0557e00640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 0x7f05440a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0548006010 tx=0x7f054802c8b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:41.981+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f055009bd30 con 0x7f0560107e00 2024-03-16T12:21:41.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:41 smithi098 ceph-mon[31633]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:41.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:41 smithi098 ceph-mon[31633]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:41.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:41 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3498260900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:41.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:41 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2352143943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.179+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f056002d1b0 con 0x7f0560107e00 2024-03-16T12:21:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.180+0000 7f0557400640 1 -- 172.21.15.98:0/402909554 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f055009b630 con 0x7f0560107e00 2024-03-16T12:21:42.161 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 msgr2=0x7f05440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 0x7f05440a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0548006010 tx=0x7f054802c8b0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 msgr2=0x7f056019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f055000e9c0 tx=0x7f055000ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 shutdown_connections 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f05440a2800 0x7f05440a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05601099b0 0x7f05601a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0560108ff0 0x7f056019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 --2- 172.21.15.98:0/402909554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0560107e00 0x7f056019e0d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 >> 172.21.15.98:0/402909554 conn(0x7f0560076da0 msgr2=0x7f056010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 shutdown_connections 2024-03-16T12:21:42.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.183+0000 7f0565400640 1 -- 172.21.15.98:0/402909554 wait complete. 2024-03-16T12:21:42.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:21:42.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:41 smithi153 ceph-mon[36901]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:42.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:41 smithi153 ceph-mon[36901]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:42.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:41 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3498260900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:42.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:41 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2352143943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:42.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:41 smithi173 ceph-mon[36632]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T12:21:42.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:41 smithi173 ceph-mon[36632]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 6 op/s 2024-03-16T12:21:42.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:41 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3498260900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:21:42.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:41 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2352143943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:21:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 -- 172.21.15.98:0/2594195754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610108ff0 msgr2=0x7f3610109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 --2- 172.21.15.98:0/2594195754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610108ff0 0x7f3610109470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009900 tx=0x7f35f802bc50 comp rx=0 tx=0).stop 2024-03-16T12:21:42.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 -- 172.21.15.98:0/2594195754 shutdown_connections 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 --2- 172.21.15.98:0/2594195754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f3610110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 --2- 172.21.15.98:0/2594195754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610108ff0 0x7f3610109470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 --2- 172.21.15.98:0/2594195754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3610107e00 0x7f3610108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 -- 172.21.15.98:0/2594195754 >> 172.21.15.98:0/2594195754 conn(0x7f3610076da0 msgr2=0x7f36100791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 -- 172.21.15.98:0/2594195754 shutdown_connections 2024-03-16T12:21:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.413+0000 7f3616000640 1 -- 172.21.15.98:0/2594195754 wait complete. 2024-03-16T12:21:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 Processor -- start 2024-03-16T12:21:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 -- start start 2024-03-16T12:21:42.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 0x7f361019e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3610108ff0 0x7f361019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f36101a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3610113ab0 con 0x7f3610108ff0 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3610113930 con 0x7f36101099b0 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.414+0000 7f3616000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3610113c30 con 0x7f3610107e00 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f36101a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 0x7f361019e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f36101a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54186/0 (socket says 172.21.15.98:54186) 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360fe00640 1 -- 172.21.15.98:0/438252723 learned_addr learned my addr 172.21.15.98:0/438252723 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 -- 172.21.15.98:0/438252723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 msgr2=0x7f36101a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f36101a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 -- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3610108ff0 msgr2=0x7f361019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3610108ff0 0x7f361019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 -- 172.21.15.98:0/438252723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36101a5d20 con 0x7f3610107e00 2024-03-16T12:21:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360f400640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 0x7f361019e100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f360000e9c0 tx=0x7f360000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360000ccc0 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36000035f0 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.415+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360000ce60 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.416+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36101a5fb0 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.416+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36101a6480 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.417+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361010f120 con 0x7f3610107e00 2024-03-16T12:21:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.418+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f36000037a0 con 0x7f3610107e00 2024-03-16T12:21:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.420+0000 7f360cc00640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 0x7f35f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.420+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f36000cf220 con 0x7f3610107e00 2024-03-16T12:21:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.420+0000 7f3606a00640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 0x7f35f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.421+0000 7f3606a00640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 0x7f35f40a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f35f8036040 tx=0x7f35f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.424+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f360009bf00 con 0x7f3610107e00 2024-03-16T12:21:42.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.623+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f36100624e0 con 0x7f3610107e00 2024-03-16T12:21:42.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.624+0000 7f360cc00640 1 -- 172.21.15.98:0/438252723 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f360009b800 con 0x7f3610107e00 2024-03-16T12:21:42.605 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 msgr2=0x7f35f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 0x7f35f40a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f35f8036040 tx=0x7f35f8000bc0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 msgr2=0x7f361019e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 0x7f361019e100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f360000e9c0 tx=0x7f360000ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 shutdown_connections 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f35f40a2800 0x7f35f40a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36101099b0 0x7f36101a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3610108ff0 0x7f361019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 --2- 172.21.15.98:0/438252723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3610107e00 0x7f361019e100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.626+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 >> 172.21.15.98:0/438252723 conn(0x7f3610076da0 msgr2=0x7f361010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.627+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 shutdown_connections 2024-03-16T12:21:42.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.627+0000 7f3616000640 1 -- 172.21.15.98:0/438252723 wait complete. 2024-03-16T12:21:42.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 -- 172.21.15.98:0/500837616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a41018c0 msgr2=0x7fb4a4101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 --2- 172.21.15.98:0/500837616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a41018c0 0x7fb4a4101d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb494009900 tx=0x7fb49402bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 -- 172.21.15.98:0/500837616 shutdown_connections 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 --2- 172.21.15.98:0/500837616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a4102280 0x7fb4a4108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 --2- 172.21.15.98:0/500837616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a41018c0 0x7fb4a4101d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 --2- 172.21.15.98:0/500837616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41006d0 0x7fb4a4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 -- 172.21.15.98:0/500837616 >> 172.21.15.98:0/500837616 conn(0x7fb4a40fbc40 msgr2=0x7fb4a40fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 -- 172.21.15.98:0/500837616 shutdown_connections 2024-03-16T12:21:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.858+0000 7fb4ac800640 1 -- 172.21.15.98:0/500837616 wait complete. 2024-03-16T12:21:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 Processor -- start 2024-03-16T12:21:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 -- start start 2024-03-16T12:21:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41018c0 0x7fb4a4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a4102280 0x7fb4a419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a410c3a0 con 0x7fb4a41018c0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a410c220 con 0x7fb4a4102280 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.859+0000 7fb4ac800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a410c520 con 0x7fb4a41006d0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41018c0 0x7fb4a4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4aa800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a4102280 0x7fb4a419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56342/0 (socket says 172.21.15.98:56342) 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 -- 172.21.15.98:0/3589390919 learned_addr learned my addr 172.21.15.98:0/3589390919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 -- 172.21.15.98:0/3589390919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a4102280 msgr2=0x7fb4a419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a4102280 0x7fb4a419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 -- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41018c0 msgr2=0x7fb4a4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41018c0 0x7fb4a4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 -- 172.21.15.98:0/3589390919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a419f700 con 0x7fb4a41006d0 2024-03-16T12:21:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.860+0000 7fb4a9e00640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb49800e9c0 tx=0x7fb49800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.861+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49800cd80 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.861+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4980035f0 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.861+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a419f9f0 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.861+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498010690 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.861+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a419ff10 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.862+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a41a01a0 con 0x7fb4a41006d0 2024-03-16T12:21:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.864+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb4980037a0 con 0x7fb4a41006d0 2024-03-16T12:21:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.865+0000 7fb49ea00640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 0x7fb48c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.865+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb4980cf170 con 0x7fb4a41006d0 2024-03-16T12:21:42.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.865+0000 7fb4a9400640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 0x7fb48c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:42.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.866+0000 7fb4a9400640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 0x7fb48c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb49402c240 tx=0x7fb49402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:42.869+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb49809beb0 con 0x7fb4a41006d0 2024-03-16T12:21:43.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.110+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4a41a0950 con 0x7fb4a41006d0 2024-03-16T12:21:43.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.111+0000 7fb49ea00640 1 -- 172.21.15.98:0/3589390919 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7fb49809b7b0 con 0x7fb4a41006d0 2024-03-16T12:21:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:43.093 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:21:43.093 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.113+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 msgr2=0x7fb48c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.113+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 0x7fb48c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb49402c240 tx=0x7fb49402df90 comp rx=0 tx=0).stop 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.113+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 msgr2=0x7fb4a4197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.113+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb49800e9c0 tx=0x7fb49800ee80 comp rx=0 tx=0).stop 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 shutdown_connections 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb48c0a2800 0x7fb48c0a4cc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4a4102280 0x7fb4a419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4a41018c0 0x7fb4a4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 --2- 172.21.15.98:0/3589390919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4a41006d0 0x7fb4a4197ae0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 >> 172.21.15.98:0/3589390919 conn(0x7fb4a40fbc40 msgr2=0x7fb4a40fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 shutdown_connections 2024-03-16T12:21:43.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.114+0000 7fb4ac800640 1 -- 172.21.15.98:0/3589390919 wait complete. 2024-03-16T12:21:43.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:21:43.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/402909554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:42 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/438252723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/402909554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:42 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/438252723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/402909554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:42 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/438252723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.338+0000 7faf06e00640 1 -- 172.21.15.98:0/3623743701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf00104bf0 msgr2=0x7faf00105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.338+0000 7faf06e00640 1 --2- 172.21.15.98:0/3623743701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf00104bf0 0x7faf00105070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faee8009900 tx=0x7faee802bc30 comp rx=0 tx=0).stop 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 -- 172.21.15.98:0/3623743701 shutdown_connections 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 --2- 172.21.15.98:0/3623743701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf001055b0 0x7faf0010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 --2- 172.21.15.98:0/3623743701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf00104bf0 0x7faf00105070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 --2- 172.21.15.98:0/3623743701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00103a00 0x7faf00103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 -- 172.21.15.98:0/3623743701 >> 172.21.15.98:0/3623743701 conn(0x7faf000fef90 msgr2=0x7faf001013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 -- 172.21.15.98:0/3623743701 shutdown_connections 2024-03-16T12:21:43.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.339+0000 7faf06e00640 1 -- 172.21.15.98:0/3623743701 wait complete. 2024-03-16T12:21:43.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 Processor -- start 2024-03-16T12:21:43.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 -- start start 2024-03-16T12:21:43.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 0x7faf00199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf001055b0 0x7faf001a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f6d0 con 0x7faf00103a00 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f550 con 0x7faf00104bf0 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.340+0000 7faf06e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f850 con 0x7faf001055b0 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faeffe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 0x7faf00199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54220/0 (socket says 172.21.15.98:54220) 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faeffe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 0x7faf00199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43238/0 (socket says 172.21.15.98:43238) 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 -- 172.21.15.98:0/3026624495 learned_addr learned my addr 172.21.15.98:0/3026624495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faf04e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf001055b0 0x7faf001a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 -- 172.21.15.98:0/3026624495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf001055b0 msgr2=0x7faf001a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf001055b0 0x7faf001a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 -- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 msgr2=0x7faf00199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 0x7faf00199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 -- 172.21.15.98:0/3026624495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf001a18d0 con 0x7faf00104bf0 2024-03-16T12:21:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.341+0000 7faef7400640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faee8005ab0 tx=0x7faee802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.342+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8043020 con 0x7faf00104bf0 2024-03-16T12:21:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.342+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee80054e0 con 0x7faf00104bf0 2024-03-16T12:21:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.342+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf001a1b60 con 0x7faf00104bf0 2024-03-16T12:21:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.342+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8003ce0 con 0x7faf00104bf0 2024-03-16T12:21:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.342+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf001a20b0 con 0x7faf00104bf0 2024-03-16T12:21:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.343+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0010ad20 con 0x7faf00104bf0 2024-03-16T12:21:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.345+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faee8046020 con 0x7faf00104bf0 2024-03-16T12:21:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.346+0000 7faefd600640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 0x7faee40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.346+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faee80f0eb0 con 0x7faf00104bf0 2024-03-16T12:21:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.346+0000 7faeffe00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 0x7faee40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.347+0000 7faeffe00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 0x7faee40a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faf0019ab60 tx=0x7faef0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.350+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faee804d020 con 0x7faf00104bf0 2024-03-16T12:21:43.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.550+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf000623e0 con 0x7faf00104bf0 2024-03-16T12:21:43.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.552+0000 7faefd600640 1 -- 172.21.15.98:0/3026624495 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7faee80bdbf0 con 0x7faf00104bf0 2024-03-16T12:21:43.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:21:43.534 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 msgr2=0x7faee40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 0x7faee40a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faf0019ab60 tx=0x7faef0005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 msgr2=0x7faf0019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faee8005ab0 tx=0x7faee802c6c0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 shutdown_connections 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7faee40a2800 0x7faee40a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf001055b0 0x7faf001a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00104bf0 0x7faf0019a1d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 --2- 172.21.15.98:0/3026624495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf00103a00 0x7faf00199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.554+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 >> 172.21.15.98:0/3026624495 conn(0x7faf000fef90 msgr2=0x7faf00107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.555+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 shutdown_connections 2024-03-16T12:21:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.555+0000 7faf06e00640 1 -- 172.21.15.98:0/3026624495 wait complete. 2024-03-16T12:21:43.556 DEBUG:tasks.fwd_scrub:Yielding 2024-03-16T12:21:43.556 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T12:21:43.571 INFO:tasks.workunit:Pulling workunits from ref 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:21:43.571 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T12:21:43.571 INFO:tasks.workunit:timeout=3h 2024-03-16T12:21:43.571 INFO:tasks.workunit:cleanup=True 2024-03-16T12:21:43.571 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:43.572 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-16T12:21:43.572 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T12:21:43.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:21:43.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T12:21:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-16T12:21:43.618 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-16T12:21:43.618 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T12:21:43.618 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-16T12:21:43.619 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-16 12:19:09.467267941 +0000 2024-03-16T12:21:43.619 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-16 12:19:09.467267941 +0000 2024-03-16T12:21:43.619 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-16 12:21:32.526478252 +0000 2024-03-16T12:21:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-16 12:19:09.467267941 +0000 2024-03-16T12:21:43.619 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T12:21:43.619 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T12:21:43.756 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T12:21:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 -- 172.21.15.98:0/3098073291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 msgr2=0x7f5554100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 --2- 172.21.15.98:0/3098073291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554100b10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5544009ca0 tx=0x7f554402c100 comp rx=0 tx=0).stop 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 -- 172.21.15.98:0/3098073291 shutdown_connections 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 --2- 172.21.15.98:0/3098073291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541022a0 0x7f5554108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 --2- 172.21.15.98:0/3098073291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541018e0 0x7f5554101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 --2- 172.21.15.98:0/3098073291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554100b10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 -- 172.21.15.98:0/3098073291 >> 172.21.15.98:0/3098073291 conn(0x7f55540fbc80 msgr2=0x7f55540fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 -- 172.21.15.98:0/3098073291 shutdown_connections 2024-03-16T12:21:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.791+0000 7f5559a00640 1 -- 172.21.15.98:0/3098073291 wait complete. 2024-03-16T12:21:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 Processor -- start 2024-03-16T12:21:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 -- start start 2024-03-16T12:21:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541018e0 0x7f555419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541022a0 0x7f55541a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f555410c370 con 0x7f55541022a0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f555410c1f0 con 0x7f55541018e0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.792+0000 7f5559a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f555410c4f0 con 0x7f55541006f0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5553400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541022a0 0x7f55541a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56370/0 (socket says 172.21.15.98:56370) 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 -- 172.21.15.98:0/4071204021 learned_addr learned my addr 172.21.15.98:0/4071204021 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f554a000640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541018e0 0x7f555419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 -- 172.21.15.98:0/4071204021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541018e0 msgr2=0x7f555419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541018e0 0x7f555419a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 -- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541022a0 msgr2=0x7f55541a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541022a0 0x7f55541a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 -- 172.21.15.98:0/4071204021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55541a1980 con 0x7f55541006f0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.793+0000 7f5552a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5544000f80 tx=0x7f5544035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.794+0000 7f554be00640 1 -- 172.21.15.98:0/4071204021 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544039040 con 0x7f55541006f0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.794+0000 7f554be00640 1 -- 172.21.15.98:0/4071204021 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5544044070 con 0x7f55541006f0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.794+0000 7f554be00640 1 -- 172.21.15.98:0/4071204021 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544004430 con 0x7f55541006f0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.794+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55541a1c10 con 0x7f55541006f0 2024-03-16T12:21:43.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.794+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55541a2080 con 0x7f55541006f0 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 Processor -- start 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 -- start start 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f554be00640 1 -- 172.21.15.98:0/4071204021 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f55440045d0 con 0x7f55541006f0 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948c1d0 0x557a29490cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a293eef80 con 0x557a2948c1d0 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557a29397ad0 con 0x557a29491340 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.797+0000 7f555c7c6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557a293eb020 con 0x557a2948b7e0 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f554a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56378/0 (socket says 172.21.15.98:56378) 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 -- 172.21.15.98:0/4142624465 learned_addr learned my addr 172.21.15.98:0/4142624465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f554a000640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54238/0 (socket says 172.21.15.98:54238) 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5552a00640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948c1d0 0x557a29490cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 -- 172.21.15.98:0/4142624465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 msgr2=0x557a294937d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 -- 172.21.15.98:0/4142624465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948c1d0 msgr2=0x557a29490cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948c1d0 0x557a29490cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 -- 172.21.15.98:0/4142624465 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a293af6f0 con 0x557a2948b7e0 2024-03-16T12:21:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f554a000640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:43.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5553400640 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f554c009ca0 tx=0x7f554c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be2e3c34a22f5697 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5543e00640 1 -- 172.21.15.98:0/4142624465 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c01e070 con 0x557a2948b7e0 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.798+0000 7f5543e00640 1 -- 172.21.15.98:0/4142624465 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f554c009e50 con 0x557a2948b7e0 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f5543e00640 1 -- 172.21.15.98:0/4142624465 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c00eac0 con 0x557a2948b7e0 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f554be00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 0x7f55380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f554be00640 1 -- 172.21.15.98:0/4071204021 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55440f1260 con 0x7f55541006f0 2024-03-16T12:21:43.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f5552a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 0x7f55380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f5552a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 0x7f55380a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5544006010 tx=0x7f55440095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f555c7c6740 1 -- 172.21.15.98:0/4142624465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 msgr2=0x557a2948bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.799+0000 7f555c7c6740 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f554c009ca0 tx=0x7f554c011810 comp rx=0 tx=0).stop 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 -- 172.21.15.98:0/4142624465 shutdown_connections 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a294937d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948c1d0 0x557a29490cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 --2- 172.21.15.98:0/4142624465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948b7e0 0x557a2948bc00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 -- 172.21.15.98:0/4142624465 >> 172.21.15.98:0/4142624465 conn(0x557a29453930 msgr2=0x557a29487150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 -- 172.21.15.98:0/4142624465 shutdown_connections 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 -- 172.21.15.98:0/4142624465 wait complete. 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 -- start start 2024-03-16T12:21:43.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.800+0000 7f555c7c6740 1 Processor -- start 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a29495b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a2949d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557a294942f0 con 0x557a2948b7e0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557a293aaf30 con 0x557a29491340 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557a29494470 con 0x557a2948c1d0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f555c7c6740 10 client.0 _ll_register_callbacks cb 0x557a29398bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f5553400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a29495b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56392/0 (socket says 172.21.15.98:56392) 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 -- 172.21.15.98:0/2834925160 learned_addr learned my addr 172.21.15.98:0/2834925160 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f5553400640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a2949d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 -- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 msgr2=0x557a2949d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a2949d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 -- 172.21.15.98:0/2834925160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 msgr2=0x557a29495b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a29495b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.801+0000 7f554a000640 1 -- 172.21.15.98:0/2834925160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a291cb7e0 con 0x557a2948c1d0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f554a000640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f553c012700 tx=0x7f553c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=565186712b5e5073 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f5540c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c004030 con 0x557a2948c1d0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f553c0041d0 con 0x557a2948c1d0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f554c009e50 con 0x557a2948c1d0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f553c025050 con 0x557a2948c1d0 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f555c7c6740 10 client.25528 fetch_fsmap learned FSMap version 24 2024-03-16T12:21:43.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557a2922bf70 con 0x557a2948c1d0 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.802+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f554c009e50 con 0x557a2948c1d0 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f553c003a30 con 0x557a2948c1d0 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 10 client.25528 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 20 client.25528 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a294a1a90 0x557a294a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 4 client.25528 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a293fc4c0 con 0x557a294a1a90 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.803+0000 7f5553400640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a294a1a90 0x557a294a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.804+0000 7f5553400640 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a294a1a90 0x557a294a3f10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54739c5e9270ec0a server_cookie=102da13906b117b9 in_seq=0 out_seq=0 2024-03-16T12:21:43.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.804+0000 7f5540c00640 10 client.25528 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:43.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.804+0000 7f5540c00640 1 -- 172.21.15.98:0/2834925160 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a293fc4c0 con 0x557a294a1a90 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.804+0000 7f5540c00640 10 client.25528 handle_command_reply: tid=0 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.804+0000 7f555c7c6740 1 client.25528 shutdown 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 msgr2=0x557a294960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f553c012700 tx=0x7f553c012bc0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 shutdown_connections 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a29491340 0x557a2949d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a294a1a90 0x557a294a3f10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a294960b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2834925160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a29495b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f5540c00640 0 client.25528 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 >> 172.21.15.98:0/2834925160 conn(0x557a29453930 msgr2=0x557a29486350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 shutdown_connections 2024-03-16T12:21:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 1 -- 172.21.15.98:0/2834925160 wait complete. 2024-03-16T12:21:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.805+0000 7f555c7c6740 20 client.25528 trim_cache size 0 max 16384 2024-03-16T12:21:43.802 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.823+0000 7f555c7c6740 1 Processor -- start 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 -- start start 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a294861d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a29491340 0x557a2949d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55541aedd0 con 0x557a2948b7e0 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55541ac6a0 con 0x557a2948c1d0 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f555c7c6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55541aa4f0 con 0x557a29491340 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f5553400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a294861d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54258/0 (socket says 172.21.15.98:54258) 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 -- 172.21.15.98:0/1259555625 learned_addr learned my addr 172.21.15.98:0/1259555625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f5553400640 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a29491340 0x557a2949d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 -- 172.21.15.98:0/1259555625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a29491340 msgr2=0x557a2949d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a29491340 0x557a2949d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 -- 172.21.15.98:0/1259555625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 msgr2=0x557a294861d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a294861d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.824+0000 7f554a000640 1 -- 172.21.15.98:0/1259555625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a29482980 con 0x557a2948c1d0 2024-03-16T12:21:43.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.825+0000 7f554a000640 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f553c005840 tx=0x7f553c010790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8831002e748f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.825+0000 7f5543400640 1 -- 172.21.15.98:0/1259555625 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c022040 con 0x557a2948c1d0 2024-03-16T12:21:43.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.825+0000 7f5543400640 1 -- 172.21.15.98:0/1259555625 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f553c005550 con 0x557a2948c1d0 2024-03-16T12:21:43.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.825+0000 7f5543400640 1 -- 172.21.15.98:0/1259555625 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c003850 con 0x557a2948c1d0 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 -- 172.21.15.98:0/1259555625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 msgr2=0x557a29486710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f553c005840 tx=0x7f553c010790 comp rx=0 tx=0).stop 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 -- 172.21.15.98:0/1259555625 shutdown_connections 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a29491340 0x557a2949d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948c1d0 0x557a29486710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 --2- 172.21.15.98:0/1259555625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a2948b7e0 0x557a294861d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 -- 172.21.15.98:0/1259555625 >> 172.21.15.98:0/1259555625 conn(0x557a29453930 msgr2=0x557a29485ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 -- 172.21.15.98:0/1259555625 shutdown_connections 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.826+0000 7f555c7c6740 1 -- 172.21.15.98:0/1259555625 wait complete. 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 -- start start 2024-03-16T12:21:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 Processor -- start 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a2949ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a29491340 0x557a2949af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55541aab60 con 0x557a29491340 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55541abfd0 con 0x557a2948b7e0 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55541ab230 con 0x557a2948c1d0 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.827+0000 7f555c7c6740 10 client.0 _ll_register_callbacks cb 0x7f553c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f554a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a2949ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54260/0 (socket says 172.21.15.98:54260) 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 -- 172.21.15.98:0/2255380852 learned_addr learned my addr 172.21.15.98:0/2255380852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a29491340 0x557a2949af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 -- 172.21.15.98:0/2255380852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 msgr2=0x557a2949ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a2949ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 -- 172.21.15.98:0/2255380852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a29491340 msgr2=0x557a2949af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a29491340 0x557a2949af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 -- 172.21.15.98:0/2255380852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a290c3170 con 0x557a2948b7e0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f554a000640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a2949ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5553400640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f554c005fd0 tx=0x7f554c00e420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea2b83b7f4fefd86 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.828+0000 7f5540c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c01e040 con 0x557a2948b7e0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f554c0037f0 con 0x557a2948b7e0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f553c005550 con 0x557a2948b7e0 2024-03-16T12:21:43.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f554c0039f0 con 0x557a2948b7e0 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 10 client.25547 fetch_fsmap learned FSMap version 24 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557a29508610 con 0x557a2948b7e0 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f553c005550 con 0x557a2948b7e0 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f554c024040 con 0x557a2948b7e0 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 10 client.25547 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 10 client.25547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 20 client.25547 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a29508d90 0x557a2950b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 4 client.25547 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.829+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557a293f6810 con 0x557a29508d90 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.830+0000 7f554a000640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a29508d90 0x557a2950b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.830+0000 7f5540c00640 10 client.25547 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.830+0000 7f554a000640 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a29508d90 0x557a2950b1b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbb375b6171f37fb server_cookie=fb543bb640eb7a31 in_seq=0 out_seq=0 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.831+0000 7f5540c00640 1 -- 172.21.15.98:0/2255380852 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557a293f6810 con 0x557a29508d90 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.831+0000 7f5540c00640 10 client.25547 handle_command_reply: tid=0 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.831+0000 7f555c7c6740 1 client.25547 shutdown 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.831+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 msgr2=0x557a2949c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.831+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f554c005fd0 tx=0x7f554c00e420 comp rx=0 tx=0).stop 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 shutdown_connections 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557a29491340 0x557a2949af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x557a29508d90 0x557a2950b1b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a2948c1d0 0x557a2949ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 --2- 172.21.15.98:0/2255380852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557a2948b7e0 0x557a2949c940 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f5540c00640 0 client.25547 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 >> 172.21.15.98:0/2255380852 conn(0x557a29453930 msgr2=0x557a29486350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 shutdown_connections 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 1 -- 172.21.15.98:0/2255380852 wait complete. 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.832+0000 7f555c7c6740 20 client.25547 trim_cache size 0 max 16384 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-03-16T12:21:43.818 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "d889c67b-76ed-46ec-ba9d-43a2a3a6b00e", 2024-03-16T12:21:43.819 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-03-16T12:21:43.819 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:21:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 msgr2=0x7f55380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 0x7f55380a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5544006010 tx=0x7f55440095c0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 msgr2=0x7f5554199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5544000f80 tx=0x7f5544035d10 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 shutdown_connections 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f55380a2880 0x7f55380a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55541022a0 0x7f55541a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55541018e0 0x7f555419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 --2- 172.21.15.98:0/4071204021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55541006f0 0x7f5554199d60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 >> 172.21.15.98:0/4071204021 conn(0x7f55540fbc80 msgr2=0x7f55540fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 shutdown_connections 2024-03-16T12:21:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:43.833+0000 7f5559a00640 1 -- 172.21.15.98:0/4071204021 wait complete. 2024-03-16T12:21:43.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:21:43.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 -- 172.21.15.98:0/1391427182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac4103a20 msgr2=0x7f0ac4103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 --2- 172.21.15.98:0/1391427182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac4103a20 0x7f0ac4103e40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009900 tx=0x7f0ab802bc50 comp rx=0 tx=0).stop 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 -- 172.21.15.98:0/1391427182 shutdown_connections 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 --2- 172.21.15.98:0/1391427182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac41055d0 0x7f0ac410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 --2- 172.21.15.98:0/1391427182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4104c10 0x7f0ac4105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 --2- 172.21.15.98:0/1391427182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac4103a20 0x7f0ac4103e40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.108+0000 7f0acbc00640 1 -- 172.21.15.98:0/1391427182 >> 172.21.15.98:0/1391427182 conn(0x7f0ac40fef90 msgr2=0x7f0ac41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.109+0000 7f0acbc00640 1 -- 172.21.15.98:0/1391427182 shutdown_connections 2024-03-16T12:21:44.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.109+0000 7f0acbc00640 1 -- 172.21.15.98:0/1391427182 wait complete. 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.109+0000 7f0acbc00640 1 Processor -- start 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 -- start start 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 0x7f0ac419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac41055d0 0x7f0ac41a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac410f6e0 con 0x7f0ac4104c10 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac410f560 con 0x7f0ac4103a20 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0acbc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac410f860 con 0x7f0ac41055d0 2024-03-16T12:21:44.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0ac9200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0ac9200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54274/0 (socket says 172.21.15.98:54274) 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0ac9200640 1 -- 172.21.15.98:0/2932580559 learned_addr learned my addr 172.21.15.98:0/2932580559 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac41055d0 0x7f0ac41a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.110+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 0x7f0ac419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 -- 172.21.15.98:0/2932580559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac41055d0 msgr2=0x7f0ac41a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac41055d0 0x7f0ac41a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 -- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 msgr2=0x7f0ac419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 0x7f0ac419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 -- 172.21.15.98:0/2932580559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac41a19c0 con 0x7f0ac4103a20 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 0x7f0ac419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac9200640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8005ab0 tx=0x7f0ab802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac1600640 1 -- 172.21.15.98:0/2932580559 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8041070 con 0x7f0ac4103a20 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac1600640 1 -- 172.21.15.98:0/2932580559 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab8004450 con 0x7f0ac4103a20 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac41a1c50 con 0x7f0ac4103a20 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0ac1600640 1 -- 172.21.15.98:0/2932580559 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8034440 con 0x7f0ac4103a20 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.111+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac41a21a0 con 0x7f0ac4103a20 2024-03-16T12:21:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.114+0000 7f0ac1600640 1 -- 172.21.15.98:0/2932580559 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ab8044070 con 0x7f0ac4103a20 2024-03-16T12:21:44.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 Processor -- start 2024-03-16T12:21:44.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 -- start start 2024-03-16T12:21:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b08d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e820a681f0 con 0x55e820b04200 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.115+0000 7f0ace853740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e820a10a80 con 0x55e820b09370 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ace853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e820a64290 con 0x55e820b03810 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56440/0 (socket says 172.21.15.98:56440) 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 -- 172.21.15.98:0/3314064848 learned_addr learned my addr 172.21.15.98:0/3314064848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9200640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b08d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 -- 172.21.15.98:0/3314064848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 msgr2=0x55e820b0b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 -- 172.21.15.98:0/3314064848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 msgr2=0x55e820b08d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b08d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 -- 172.21.15.98:0/3314064848 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8208467d0 con 0x55e820b03810 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9200640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac1600640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 0x7f0aac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.116+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0039f0 tx=0x7f0abc003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f04b9cdfa7257fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.117+0000 7f0ac1600640 1 -- 172.21.15.98:0/2932580559 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0ab80f0a40 con 0x7f0ac4103a20 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.117+0000 7f0ac9200640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 0x7f0aac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.117+0000 7f0ab6000640 1 -- 172.21.15.98:0/3314064848 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc01a470 con 0x55e820b03810 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.117+0000 7f0ab6000640 1 -- 172.21.15.98:0/3314064848 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0abc015070 con 0x55e820b03810 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.117+0000 7f0ab6000640 1 -- 172.21.15.98:0/3314064848 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc010900 con 0x55e820b03810 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- 172.21.15.98:0/3314064848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 msgr2=0x55e820b03c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0039f0 tx=0x7f0abc003eb0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ac9200640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 0x7f0aac0a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80bcfb0 tx=0x7f0ab8003f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- 172.21.15.98:0/3314064848 shutdown_connections 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 --2- 172.21.15.98:0/3314064848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b03810 0x55e820b03c30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- 172.21.15.98:0/3314064848 >> 172.21.15.98:0/3314064848 conn(0x55e820afed80 msgr2=0x55e820aff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- 172.21.15.98:0/3314064848 shutdown_connections 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- 172.21.15.98:0/3314064848 wait complete. 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 -- start start 2024-03-16T12:21:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.118+0000 7f0ace853740 1 Processor -- start 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b0e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b15300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e820b0c320 con 0x55e820b04200 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e820a23ed0 con 0x55e820b03810 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e820b0c4a0 con 0x55e820b09370 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ace853740 10 client.0 _ll_register_callbacks cb 0x55e820a5f220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54296/0 (socket says 172.21.15.98:54296) 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ac9c00640 1 -- 172.21.15.98:0/4158849638 learned_addr learned my addr 172.21.15.98:0/4158849638 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b0e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.119+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b15300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 -- 172.21.15.98:0/4158849638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 msgr2=0x55e820b15300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b15300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 -- 172.21.15.98:0/4158849638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 msgr2=0x55e820b0e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b0e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 -- 172.21.15.98:0/4158849638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e82077c3f0 con 0x55e820b03810 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009990 tx=0x7f0abc0098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee27d7c93b000f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0aaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc004a40 con 0x55e820b03810 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0abc004be0 con 0x55e820b03810 2024-03-16T12:21:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0abc015070 con 0x55e820b03810 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.120+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0abc004de0 con 0x55e820b03810 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 10 client.25559 fetch_fsmap learned FSMap version 24 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e82073c0f0 con 0x55e820b03810 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0abc015070 con 0x55e820b03810 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0abc0052b0 con 0x55e820b03810 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 10 client.25559 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 10 client.25559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 20 client.25559 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b198f0 0x55e820b1bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 4 client.25559 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e820a7bbe0 con 0x55e820b198f0 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.121+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b198f0 0x55e820b1bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.122+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b198f0 0x55e820b1bd00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29a9b7b2e41060c7 server_cookie=13d4c4fa9c4d75e7 in_seq=0 out_seq=0 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.122+0000 7f0aaaa00640 10 client.25559 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0aaaa00640 1 -- 172.21.15.98:0/4158849638 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e820a7bbe0 con 0x55e820b198f0 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0aaaa00640 10 client.25559 handle_command_reply: tid=0 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 client.25559 shutdown 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 msgr2=0x55e820b0dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009990 tx=0x7f0abc0098e0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 shutdown_connections 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b198f0 0x55e820b1bd00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b15300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b0e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 --2- 172.21.15.98:0/4158849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b0dd80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0aaaa00640 0 client.25559 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 >> 172.21.15.98:0/4158849638 conn(0x55e820afed80 msgr2=0x55e820afdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 shutdown_connections 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 1 -- 172.21.15.98:0/4158849638 wait complete. 2024-03-16T12:21:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.123+0000 7f0ace853740 20 client.25559 trim_cache size 0 max 16384 2024-03-16T12:21:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 Processor -- start 2024-03-16T12:21:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 -- start start 2024-03-16T12:21:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41aee40 con 0x55e820b03810 2024-03-16T12:21:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41ac710 con 0x55e820b09370 2024-03-16T12:21:44.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ace853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41aa560 con 0x55e820b04200 2024-03-16T12:21:44.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.142+0000 7f0ac3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56460/0 (socket says 172.21.15.98:56460) 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43348/0 (socket says 172.21.15.98:43348) 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 -- 172.21.15.98:0/317946497 learned_addr learned my addr 172.21.15.98:0/317946497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 -- 172.21.15.98:0/317946497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 msgr2=0x55e820b0c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 -- 172.21.15.98:0/317946497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 msgr2=0x55e820b0e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 -- 172.21.15.98:0/317946497 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e820af9370 con 0x55e820b04200 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55e820b1c540 tx=0x7f0ab0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5e27f94dac9752 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ab5600640 1 -- 172.21.15.98:0/317946497 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0005220 con 0x55e820b04200 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ab5600640 1 -- 172.21.15.98:0/317946497 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab0025030 con 0x55e820b04200 2024-03-16T12:21:44.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.143+0000 7f0ab5600640 1 -- 172.21.15.98:0/317946497 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0021530 con 0x55e820b04200 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 -- 172.21.15.98:0/317946497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 msgr2=0x55e820b152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55e820b1c540 tx=0x7f0ab0005f90 comp rx=0 tx=0).stop 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 -- 172.21.15.98:0/317946497 shutdown_connections 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b09370 0x55e820b0c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b04200 0x55e820b152c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 --2- 172.21.15.98:0/317946497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b03810 0x55e820b0e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 -- 172.21.15.98:0/317946497 >> 172.21.15.98:0/317946497 conn(0x55e820afed80 msgr2=0x55e820afd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 -- 172.21.15.98:0/317946497 shutdown_connections 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.144+0000 7f0ace853740 1 -- 172.21.15.98:0/317946497 wait complete. 2024-03-16T12:21:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 -- start start 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 Processor -- start 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b133e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b143a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41aabd0 con 0x55e820b04200 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41ac040 con 0x55e820b03810 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41ab2a0 con 0x55e820b09370 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ace853740 10 client.0 _ll_register_callbacks cb 0x7f0ab00095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.145+0000 7f0ac9c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54312/0 (socket says 172.21.15.98:54312) 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 -- 172.21.15.98:0/1987694691 learned_addr learned my addr 172.21.15.98:0/1987694691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:21:44.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 -- 172.21.15.98:0/1987694691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 msgr2=0x55e820b143a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b143a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 -- 172.21.15.98:0/1987694691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 msgr2=0x55e820b133e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b133e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 -- 172.21.15.98:0/1987694691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e820b14990 con 0x55e820b03810 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ac9c00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0abc012fd0 tx=0x7f0abc013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b88a36d6122b13c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0aaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc005aa0 con 0x55e820b03810 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0abc004f80 con 0x55e820b03810 2024-03-16T12:21:44.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.146+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab0025030 con 0x55e820b03810 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0abc005180 con 0x55e820b03810 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 10 client.25565 fetch_fsmap learned FSMap version 24 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e820b800f0 con 0x55e820b03810 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab0025030 con 0x55e820b03810 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0abc037040 con 0x55e820b03810 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 10 client.25565 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 20 client.25565 populate_metadata read hostname 'smithi098' 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b80a40 0x55e820b82e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 4 client.25565 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:21:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e820a75730 con 0x55e820b80a40 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.147+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b80a40 0x55e820b82e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ac3e00640 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b80a40 0x55e820b82e30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=895be5663af5d9ec server_cookie=d57730f63285b635 in_seq=0 out_seq=0 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0aaaa00640 10 client.25565 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0aaaa00640 1 -- 172.21.15.98:0/1987694691 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e820a75730 con 0x55e820b80a40 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0aaaa00640 10 client.25565 handle_command_reply: tid=0 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 client.25565 shutdown 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 msgr2=0x55e820b12ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0abc012fd0 tx=0x7f0abc013040 comp rx=0 tx=0).stop 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 shutdown_connections 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55e820b80a40 0x55e820b82e30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e820b09370 0x55e820b143a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e820b04200 0x55e820b133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0aaaa00640 0 client.25565 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 --2- 172.21.15.98:0/1987694691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e820b03810 0x55e820b12ea0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 >> 172.21.15.98:0/1987694691 conn(0x55e820afed80 msgr2=0x55e820afdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.148+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 shutdown_connections 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0ace853740 1 -- 172.21.15.98:0/1987694691 wait complete. 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0ace853740 20 client.25565 trim_cache size 0 max 16384 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout: "d889c67b-76ed-46ec-ba9d-43a2a3a6b00e": { 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-03-16T12:21:44.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "d889c67b-76ed-46ec-ba9d-43a2a3a6b00e", 2024-03-16T12:21:44.136 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-03-16T12:21:44.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:21:44.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:21:44.136 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 msgr2=0x7f0aac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 0x7f0aac0a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80bcfb0 tx=0x7f0ab8003f90 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 msgr2=0x7f0ac4199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8005ab0 tx=0x7f0ab802c6e0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 shutdown_connections 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0aac0a2880 0x7f0aac0a4d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac41055d0 0x7f0ac41a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac4104c10 0x7f0ac419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 --2- 172.21.15.98:0/2932580559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ac4103a20 0x7f0ac4199db0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 >> 172.21.15.98:0/2932580559 conn(0x7f0ac40fef90 msgr2=0x7f0ac4107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.149+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 shutdown_connections 2024-03-16T12:21:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:21:44.150+0000 7f0acbc00640 1 -- 172.21.15.98:0/2932580559 wait complete. 2024-03-16T12:21:44.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:43 smithi098 ceph-mon[31633]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:44.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:43 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3589390919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:44.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:43 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3026624495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:44.154 INFO:tasks.cephfs.filesystem:scrub status for tag:d889c67b-76ed-46ec-ba9d-43a2a3a6b00e - {'path': '/', 'tag': 'd889c67b-76ed-46ec-ba9d-43a2a3a6b00e', 'options': 'recursive,force'} 2024-03-16T12:21:44.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:43 smithi153 ceph-mon[36901]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:44.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:43 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3589390919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:44.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:43 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3026624495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:43 smithi173 ceph-mon[36632]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-16T12:21:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:43 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3589390919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:21:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:43 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3026624495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:21:45.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:44 smithi098 ceph-mon[31633]: scrub queued for path: / 2024-03-16T12:21:45.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:44 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:45.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:44 smithi098 ceph-mon[31633]: scrub summary: active paths [/] 2024-03-16T12:21:45.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:44 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:45.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:44 smithi153 ceph-mon[36901]: scrub queued for path: / 2024-03-16T12:21:45.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:44 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:45.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:44 smithi153 ceph-mon[36901]: scrub summary: active paths [/] 2024-03-16T12:21:45.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:44 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:45.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:44 smithi173 ceph-mon[36632]: scrub queued for path: / 2024-03-16T12:21:45.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:44 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:45.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:44 smithi173 ceph-mon[36632]: scrub summary: active paths [/] 2024-03-16T12:21:45.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:44 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:21:46.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:45 smithi098 ceph-mon[31633]: pgmap v412: 129 pgs: 129 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T12:21:46.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:45 smithi153 ceph-mon[36901]: pgmap v412: 129 pgs: 129 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T12:21:46.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:45 smithi173 ceph-mon[36632]: pgmap v412: 129 pgs: 129 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T12:21:48.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:47 smithi098 ceph-mon[31633]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-16T12:21:48.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:47 smithi153 ceph-mon[36901]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-16T12:21:48.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:47 smithi173 ceph-mon[36632]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-16T12:21:50.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:49 smithi098 ceph-mon[31633]: pgmap v414: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:50.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:49 smithi153 ceph-mon[36901]: pgmap v414: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:50.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:49 smithi173 ceph-mon[36632]: pgmap v414: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:52.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:51 smithi098 ceph-mon[31633]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:52.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:51 smithi153 ceph-mon[36901]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:51 smithi173 ceph-mon[36632]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 6 op/s 2024-03-16T12:21:54.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:53 smithi098 ceph-mon[31633]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:54.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:53 smithi153 ceph-mon[36901]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:54.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:53 smithi173 ceph-mon[36632]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:55.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:55 smithi098 ceph-mon[31633]: scrub summary: idle 2024-03-16T12:21:55.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:55 smithi153 ceph-mon[36901]: scrub summary: idle 2024-03-16T12:21:55.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:55 smithi173 ceph-mon[36632]: scrub summary: idle 2024-03-16T12:21:56.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:56 smithi098 ceph-mon[31633]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:56.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:56 smithi153 ceph-mon[36901]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:56.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:56 smithi173 ceph-mon[36632]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 7 op/s 2024-03-16T12:21:58.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:21:58 smithi098 ceph-mon[31633]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T12:21:58.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:21:58 smithi153 ceph-mon[36901]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T12:21:58.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:21:58 smithi173 ceph-mon[36632]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-16T12:22:00.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:00 smithi153 ceph-mon[36901]: pgmap v419: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T12:22:00.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:00 smithi173 ceph-mon[36632]: pgmap v419: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T12:22:00.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:00 smithi098 ceph-mon[31633]: pgmap v419: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-16T12:22:02.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:02 smithi153 ceph-mon[36901]: pgmap v420: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:02.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:02 smithi173 ceph-mon[36632]: pgmap v420: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:02.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:02 smithi098 ceph-mon[31633]: pgmap v420: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:04.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:04 smithi173 ceph-mon[36632]: pgmap v421: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:04.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:04 smithi098 ceph-mon[31633]: pgmap v421: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:04.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:04 smithi153 ceph-mon[36901]: pgmap v421: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:06.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:06 smithi153 ceph-mon[36901]: pgmap v422: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:06.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:06 smithi173 ceph-mon[36632]: pgmap v422: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:06.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:06 smithi098 ceph-mon[31633]: pgmap v422: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:08.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:08 smithi173 ceph-mon[36632]: pgmap v423: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:08.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:08 smithi098 ceph-mon[31633]: pgmap v423: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:08.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:08 smithi153 ceph-mon[36901]: pgmap v423: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:10.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:10 smithi173 ceph-mon[36632]: pgmap v424: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:10.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:10 smithi098 ceph-mon[31633]: pgmap v424: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:10.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:10 smithi153 ceph-mon[36901]: pgmap v424: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:12.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:12 smithi173 ceph-mon[36632]: pgmap v425: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:12.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:12 smithi098 ceph-mon[31633]: pgmap v425: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:12.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:12 smithi153 ceph-mon[36901]: pgmap v425: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:14.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:22:14.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:22:14.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:14 smithi173 ceph-mon[36632]: pgmap v426: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:14.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 -- 172.21.15.98:0/358669588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 msgr2=0x7f94e8073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 --2- 172.21.15.98:0/358669588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e8073c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009ca0 tx=0x7f94d802c100 comp rx=0 tx=0).stop 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 -- 172.21.15.98:0/358669588 shutdown_connections 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 --2- 172.21.15.98:0/358669588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 --2- 172.21.15.98:0/358669588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 --2- 172.21.15.98:0/358669588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e8073c70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 -- 172.21.15.98:0/358669588 >> 172.21.15.98:0/358669588 conn(0x7f94e80fcd60 msgr2=0x7f94e80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 -- 172.21.15.98:0/358669588 shutdown_connections 2024-03-16T12:22:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.404+0000 7f94ef200640 1 -- 172.21.15.98:0/358669588 wait complete. 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.405+0000 7f94ef200640 1 Processor -- start 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.405+0000 7f94ef200640 1 -- start start 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8113a60 con 0x7f94e8074240 2024-03-16T12:22:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81138e0 con 0x7f94e810ddc0 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ef200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8113be0 con 0x7f94e8075820 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:52934/0 (socket says 172.21.15.98:52934) 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53118/0 (socket says 172.21.15.98:53118) 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7e00640 1 -- 172.21.15.98:0/3869447551 learned_addr learned my addr 172.21.15.98:0/3869447551 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57600/0 (socket says 172.21.15.98:57600) 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 -- 172.21.15.98:0/3869447551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 msgr2=0x7f94e81a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 -- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 msgr2=0x7f94e819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.406+0000 7f94e7400640 1 -- 172.21.15.98:0/3869447551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e81a5d20 con 0x7f94e8075820 2024-03-16T12:22:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94e7400640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94e4c00640 1 -- 172.21.15.98:0/3869447551 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ccc0 con 0x7f94e8075820 2024-03-16T12:22:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94e4c00640 1 -- 172.21.15.98:0/3869447551 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d4010910 con 0x7f94e8075820 2024-03-16T12:22:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94e4c00640 1 -- 172.21.15.98:0/3869447551 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4018960 con 0x7f94e8075820 2024-03-16T12:22:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e81a5fb0 con 0x7f94e8075820 2024-03-16T12:22:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.407+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e81a6480 con 0x7f94e8075820 2024-03-16T12:22:14.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.410+0000 7f94f1fe1740 1 Processor -- start 2024-03-16T12:22:14.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.410+0000 7f94f1fe1740 1 -- start start 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba490ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 0x560ba4911800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560ba486e1f0 con 0x560ba490f370 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560ba4816a80 con 0x560ba490a200 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ba486a290 con 0x560ba4909810 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94e4c00640 1 -- 172.21.15.98:0/3869447551 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94d4010ac0 con 0x7f94e8075820 2024-03-16T12:22:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57610/0 (socket says 172.21.15.98:57610) 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba490ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 -- 172.21.15.98:0/3212290705 learned_addr learned my addr 172.21.15.98:0/3212290705 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94e7400640 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 0x560ba4911800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 -- 172.21.15.98:0/3212290705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 msgr2=0x560ba490ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba490ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 -- 172.21.15.98:0/3212290705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 msgr2=0x560ba4911800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 0x560ba4911800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94ed200640 1 -- 172.21.15.98:0/3212290705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba464c7d0 con 0x560ba4909810 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.411+0000 7f94e7400640 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 0x560ba4911800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94ed200640 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009ca0 tx=0x7f94e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7dd9731ea700a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94dd600640 1 -- 172.21.15.98:0/3212290705 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e001e070 con 0x560ba4909810 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94dd600640 1 -- 172.21.15.98:0/3212290705 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e0009e50 con 0x560ba4909810 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94dd600640 1 -- 172.21.15.98:0/3212290705 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e000ea60 con 0x560ba4909810 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94e4c00640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 0x7f94cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94e4c00640 1 -- 172.21.15.98:0/3869447551 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f94d40cf0b0 con 0x7f94e8075820 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3212290705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 msgr2=0x560ba4909c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.412+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009ca0 tx=0x7f94e0011810 comp rx=0 tx=0).stop 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94e7e00640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 0x7f94cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3212290705 shutdown_connections 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490f370 0x560ba4911800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba490ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3212290705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4909c30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3212290705 >> 172.21.15.98:0/3212290705 conn(0x560ba4904d80 msgr2=0x560ba4905180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3212290705 shutdown_connections 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3212290705 wait complete. 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94f1fe1740 1 -- start start 2024-03-16T12:22:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.413+0000 7f94e7e00640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 0x7f94cc0a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f94d8004af0 tx=0x7f94d80325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 Processor -- start 2024-03-16T12:22:14.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba4913d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 0x560ba4914260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560ba4912320 con 0x560ba4909810 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560ba4829ed0 con 0x560ba490f370 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ba49124a0 con 0x560ba490a200 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94f1fe1740 10 client.0 _ll_register_callbacks cb 0x560ba4865220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba4913d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94e7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 0x560ba4914260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba4913d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53126/0 (socket says 172.21.15.98:53126) 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94ed200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:52954/0 (socket says 172.21.15.98:52954) 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.414+0000 7f94e7e00640 1 -- 172.21.15.98:0/3440494223 learned_addr learned my addr 172.21.15.98:0/3440494223 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 -- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 msgr2=0x560ba491b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 -- 172.21.15.98:0/3440494223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 msgr2=0x560ba4913d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba4913d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 -- 172.21.15.98:0/3440494223 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba45823f0 con 0x560ba490a200 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94ed200640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94e7400640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 0x560ba4914260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94d40cc8a0 tx=0x7f94d409b100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac7f8c8f21e1868 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94d2000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d40a3e90 con 0x560ba490a200 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f94d409f400 con 0x560ba490a200 2024-03-16T12:22:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94e0009e50 con 0x560ba490a200 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f94d409f620 con 0x560ba490a200 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94f1fe1740 10 client.25564 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.415+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560ba45420f0 con 0x560ba490a200 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e0009e50 con 0x560ba490a200 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f94d409abf0 con 0x560ba490a200 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 10 client.25564 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 10 client.25564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 20 client.25564 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba491fb20 0x560ba4921f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 4 client.25564 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ba4881be0 con 0x560ba491fb20 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.416+0000 7f94e7e00640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba491fb20 0x560ba4921f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.417+0000 7f94e7e00640 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba491fb20 0x560ba4921f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17040a07b883737 server_cookie=ba6863e1cd4c51ba in_seq=0 out_seq=0 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.417+0000 7f94d2000640 10 client.25564 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94d2000640 1 -- 172.21.15.98:0/3440494223 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ba4881be0 con 0x560ba491fb20 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94d2000640 10 client.25564 handle_command_reply: tid=0 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 client.25564 shutdown 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 msgr2=0x560ba4914260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 0x560ba4914260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94d40cc8a0 tx=0x7f94d409b100 comp rx=0 tx=0).stop 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 shutdown_connections 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba491b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba491fb20 0x560ba4921f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490a200 0x560ba4914260 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/3440494223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba4913d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94d2000640 0 client.25564 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 >> 172.21.15.98:0/3440494223 conn(0x560ba4904d80 msgr2=0x560ba4903ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 shutdown_connections 2024-03-16T12:22:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.418+0000 7f94f1fe1740 1 -- 172.21.15.98:0/3440494223 wait complete. 2024-03-16T12:22:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.419+0000 7f94f1fe1740 20 client.25564 trim_cache size 0 max 16384 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.437+0000 7f94f1fe1740 1 Processor -- start 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.437+0000 7f94f1fe1740 1 -- start start 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490a200 0x560ba491b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba4926850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81b3260 con 0x560ba490a200 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81b0b30 con 0x560ba490f370 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81ae980 con 0x560ba4909810 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490a200 0x560ba491b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57618/0 (socket says 172.21.15.98:57618) 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 -- 172.21.15.98:0/4251758503 learned_addr learned my addr 172.21.15.98:0/4251758503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94ed200640 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba4926850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 -- 172.21.15.98:0/4251758503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 msgr2=0x560ba4926850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba4926850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 -- 172.21.15.98:0/4251758503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490a200 msgr2=0x560ba491b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490a200 0x560ba491b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.438+0000 7f94e7e00640 1 -- 172.21.15.98:0/4251758503 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba48ff370 con 0x560ba4909810 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94e7e00640 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f94d8000f80 tx=0x7f94d8045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=827c35a9c75f90e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94dcc00640 1 -- 172.21.15.98:0/4251758503 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8059070 con 0x560ba4909810 2024-03-16T12:22:14.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94dcc00640 1 -- 172.21.15.98:0/4251758503 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d803a040 con 0x560ba4909810 2024-03-16T12:22:14.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94dcc00640 1 -- 172.21.15.98:0/4251758503 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8039040 con 0x560ba4909810 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94f1fe1740 1 -- 172.21.15.98:0/4251758503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 msgr2=0x560ba4903d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.439+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f94d8000f80 tx=0x7f94d8045000 comp rx=0 tx=0).stop 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 -- 172.21.15.98:0/4251758503 shutdown_connections 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490f370 0x560ba4926850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba490a200 0x560ba491b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/4251758503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba4909810 0x560ba4903d40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 -- 172.21.15.98:0/4251758503 >> 172.21.15.98:0/4251758503 conn(0x560ba4904d80 msgr2=0x560ba4903630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 -- 172.21.15.98:0/4251758503 shutdown_connections 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 -- 172.21.15.98:0/4251758503 wait complete. 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 -- start start 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.440+0000 7f94f1fe1740 1 Processor -- start 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba4919590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 0x560ba4919ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81aeff0 con 0x560ba4909810 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81b0460 con 0x560ba490a200 2024-03-16T12:22:14.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81af6c0 con 0x560ba490f370 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94f1fe1740 10 client.0 _ll_register_callbacks cb 0x7f94d400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94ed200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 0x560ba4919ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba4919590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53148/0 (socket says 172.21.15.98:53148) 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94ed200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 0x560ba4919ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57628/0 (socket says 172.21.15.98:57628) 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7e00640 1 -- 172.21.15.98:0/2621449007 learned_addr learned my addr 172.21.15.98:0/2621449007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 -- 172.21.15.98:0/2621449007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 msgr2=0x560ba4919ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 0x560ba4919ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 -- 172.21.15.98:0/2621449007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 msgr2=0x560ba491af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7400640 1 -- 172.21.15.98:0/2621449007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba491a110 con 0x560ba490a200 2024-03-16T12:22:14.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.441+0000 7f94e7e00640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94e7400640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba4919590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f94d4010f50 tx=0x7f94d4099cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d680d081ebf23818 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d40a3e90 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f94d40d2390 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d803a040 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f94d409b570 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 10 client.25583 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560ba4986d80 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d803a040 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f94d4018b00 con 0x560ba490a200 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 10 client.25583 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 20 client.25583 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:14.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.442+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba49875a0 0x560ba49899c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94f1fe1740 4 client.25583 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ba487b730 con 0x560ba49875a0 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94e7e00640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba49875a0 0x560ba49899c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94e7e00640 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba49875a0 0x560ba49899c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f764719fe953a37 server_cookie=9bdefdd5aa5c4d4a in_seq=0 out_seq=0 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94d2000640 10 client.25583 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94d2000640 1 -- 172.21.15.98:0/2621449007 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560ba487b730 con 0x560ba49875a0 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94d2000640 10 client.25583 handle_command_reply: tid=0 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.443+0000 7f94f1fe1740 1 client.25583 shutdown 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 msgr2=0x560ba4919590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba4919590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f94d4010f50 tx=0x7f94d4099cb0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 shutdown_connections 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x560ba49875a0 0x560ba49899c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ba490f370 0x560ba4919ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94d2000640 0 client.25583 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ba490a200 0x560ba4919590 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 --2- 172.21.15.98:0/2621449007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ba4909810 0x560ba491af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 >> 172.21.15.98:0/2621449007 conn(0x560ba4904d80 msgr2=0x560ba4903ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 shutdown_connections 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 1 -- 172.21.15.98:0/2621449007 wait complete. 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94f1fe1740 20 client.25583 trim_cache size 0 max 16384 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-03-16T12:22:14.431 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 msgr2=0x7f94cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.444+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 0x7f94cc0a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f94d8004af0 tx=0x7f94d80325b0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 msgr2=0x7f94e819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 shutdown_connections 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f94cc0a2880 0x7f94cc0a4d40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e810ddc0 0x7f94e81a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94e8075820 0x7f94e819e640 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 --2- 172.21.15.98:0/3869447551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94e8074240 0x7f94e819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 >> 172.21.15.98:0/3869447551 conn(0x7f94e80fcd60 msgr2=0x7f94e8101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 shutdown_connections 2024-03-16T12:22:14.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.445+0000 7f94ef200640 1 -- 172.21.15.98:0/3869447551 wait complete. 2024-03-16T12:22:14.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:14 smithi098 ceph-mon[31633]: pgmap v426: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:14.445 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T12:22:14.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:22:14.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:14 smithi153 ceph-mon[36901]: pgmap v426: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.678+0000 7fc6e2000640 1 -- 172.21.15.98:0/3431993988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 msgr2=0x7fc6dc110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.678+0000 7fc6e2000640 1 --2- 172.21.15.98:0/3431993988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc110060 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009ca0 tx=0x7fc6d402c100 comp rx=0 tx=0).stop 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.678+0000 7fc6e2000640 1 -- 172.21.15.98:0/3431993988 shutdown_connections 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.678+0000 7fc6e2000640 1 --2- 172.21.15.98:0/3431993988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc110060 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.678+0000 7fc6e2000640 1 --2- 172.21.15.98:0/3431993988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.679+0000 7fc6e2000640 1 --2- 172.21.15.98:0/3431993988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.679+0000 7fc6e2000640 1 -- 172.21.15.98:0/3431993988 >> 172.21.15.98:0/3431993988 conn(0x7fc6dc076bb0 msgr2=0x7fc6dc078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.679+0000 7fc6e2000640 1 -- 172.21.15.98:0/3431993988 shutdown_connections 2024-03-16T12:22:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.679+0000 7fc6e2000640 1 -- 172.21.15.98:0/3431993988 wait complete. 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 Processor -- start 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 -- start start 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc1138a0 con 0x7fc6dc107c20 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc113720 con 0x7fc6dc108e10 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.680+0000 7fc6e2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc113a20 con 0x7fc6dc1097d0 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6e0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:52984/0 (socket says 172.21.15.98:52984) 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6e0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53166/0 (socket says 172.21.15.98:53166) 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 -- 172.21.15.98:0/1637452966 learned_addr learned my addr 172.21.15.98:0/1637452966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6e1600640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 -- 172.21.15.98:0/1637452966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 msgr2=0x7fc6dc1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 -- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 msgr2=0x7fc6dc19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 -- 172.21.15.98:0/1637452966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6dc1a5aa0 con 0x7fc6dc108e10 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6e1600640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6e0c00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.681+0000 7fc6d3e00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400e9f0 tx=0x7fc6c400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.682+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400cd80 con 0x7fc6dc108e10 2024-03-16T12:22:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.682+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c4005500 con 0x7fc6dc108e10 2024-03-16T12:22:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.682+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4010690 con 0x7fc6dc108e10 2024-03-16T12:22:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.682+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6dc1a5d90 con 0x7fc6dc108e10 2024-03-16T12:22:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.682+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6dc1a6230 con 0x7fc6dc108e10 2024-03-16T12:22:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.683+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6dc10eee0 con 0x7fc6dc108e10 2024-03-16T12:22:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.685+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc6c40046c0 con 0x7fc6dc108e10 2024-03-16T12:22:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.686+0000 7fc6da000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 0x7fc6c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.686+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc6c40cf200 con 0x7fc6dc108e10 2024-03-16T12:22:14.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.686+0000 7fc6e0c00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 0x7fc6c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.687+0000 7fc6e0c00640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 0x7fc6c00a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc108c70 tx=0x7fc6cc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:14.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.694+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6c409bee0 con 0x7fc6dc108e10 2024-03-16T12:22:14.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.934+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6dc02d1b0 con 0x7fc6dc108e10 2024-03-16T12:22:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.935+0000 7fc6da000640 1 -- 172.21.15.98:0/1637452966 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7fc6c409b7e0 con 0x7fc6dc108e10 2024-03-16T12:22:14.916 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:22:14.916 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:22:14.916 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.937+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 msgr2=0x7fc6c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.937+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 0x7fc6c00a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc108c70 tx=0x7fc6cc00b390 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.937+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 msgr2=0x7fc6dc19e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.937+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400e9f0 tx=0x7fc6c400edf0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 shutdown_connections 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc6c00a2800 0x7fc6c00a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6dc1097d0 0x7fc6dc1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6dc108e10 0x7fc6dc19e3c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 --2- 172.21.15.98:0/1637452966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6dc107c20 0x7fc6dc19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 >> 172.21.15.98:0/1637452966 conn(0x7fc6dc076bb0 msgr2=0x7fc6dc10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 shutdown_connections 2024-03-16T12:22:14.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:14.938+0000 7fc6e2000640 1 -- 172.21.15.98:0/1637452966 wait complete. 2024-03-16T12:22:14.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T12:22:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 -- 172.21.15.98:0/717999217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0103a10 msgr2=0x7fafc0103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 --2- 172.21.15.98:0/717999217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0103a10 0x7fafc0103e30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009900 tx=0x7fafb002bc40 comp rx=0 tx=0).stop 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 -- 172.21.15.98:0/717999217 shutdown_connections 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 --2- 172.21.15.98:0/717999217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 0x7fafc010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 --2- 172.21.15.98:0/717999217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0104c00 0x7fafc0105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 --2- 172.21.15.98:0/717999217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0103a10 0x7fafc0103e30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 -- 172.21.15.98:0/717999217 >> 172.21.15.98:0/717999217 conn(0x7fafc00fefa0 msgr2=0x7fafc01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 -- 172.21.15.98:0/717999217 shutdown_connections 2024-03-16T12:22:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.175+0000 7fafc5a00640 1 -- 172.21.15.98:0/717999217 wait complete. 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.176+0000 7fafc5a00640 1 Processor -- start 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.176+0000 7fafc5a00640 1 -- start start 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 0x7fafc0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 0x7fafc01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010f6d0 con 0x7fafc01055c0 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010f550 con 0x7fafc0103a10 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafc5a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010f850 con 0x7fafc0104c00 2024-03-16T12:22:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57672/0 (socket says 172.21.15.98:57672) 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 -- 172.21.15.98:0/1236563917 learned_addr learned my addr 172.21.15.98:0/1236563917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 0x7fafc0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbf400640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 0x7fafc01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 -- 172.21.15.98:0/1236563917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 msgr2=0x7fafc0199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 0x7fafc0199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 -- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 msgr2=0x7fafc01a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 0x7fafc01a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbe000640 1 -- 172.21.15.98:0/1236563917 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc01a1940 con 0x7fafc0104c00 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.177+0000 7fafbea00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 0x7fafc0199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafbe000640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e9f0 tx=0x7fafac00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafb7400640 1 -- 172.21.15.98:0/1236563917 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac00cd80 con 0x7fafc0104c00 2024-03-16T12:22:15.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafb7400640 1 -- 172.21.15.98:0/1236563917 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafac005500 con 0x7fafc0104c00 2024-03-16T12:22:15.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafb7400640 1 -- 172.21.15.98:0/1236563917 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac010690 con 0x7fafc0104c00 2024-03-16T12:22:15.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc01a1c30 con 0x7fafc0104c00 2024-03-16T12:22:15.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.178+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc01a2100 con 0x7fafc0104c00 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.181+0000 7fafc87ab740 1 Processor -- start 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.181+0000 7fafc87ab740 1 -- start start 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafb7400640 1 -- 172.21.15.98:0/1236563917 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafac0046c0 con 0x7fafc0104c00 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d22b0 0x5642fc8d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d7420 0x5642fc8d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5642fc8993b0 con 0x5642fc8d7420 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5642fc836450 con 0x5642fc8d18c0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafc87ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5642fc7decb0 con 0x5642fc8d22b0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53022/0 (socket says 172.21.15.98:53022) 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 -- 172.21.15.98:0/793020017 learned_addr learned my addr 172.21.15.98:0/793020017 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbea00640 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d22b0 0x5642fc8d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbe000640 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d7420 0x5642fc8d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 -- 172.21.15.98:0/793020017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d22b0 msgr2=0x5642fc8d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d22b0 0x5642fc8d6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 -- 172.21.15.98:0/793020017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d7420 msgr2=0x5642fc8d98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d7420 0x5642fc8d98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.182+0000 7fafbf400640 1 -- 172.21.15.98:0/793020017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642fc8001f0 con 0x5642fc8d18c0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafbf400640 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009fd0 tx=0x7fafb80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76bc12d69002f78a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafabe00640 1 -- 172.21.15.98:0/793020017 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8004d20 con 0x5642fc8d18c0 2024-03-16T12:22:15.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafabe00640 1 -- 172.21.15.98:0/793020017 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb8004050 con 0x5642fc8d18c0 2024-03-16T12:22:15.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafabe00640 1 -- 172.21.15.98:0/793020017 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8005260 con 0x5642fc8d18c0 2024-03-16T12:22:15.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafb7400640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 0x7fafa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafb7400640 1 -- 172.21.15.98:0/1236563917 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fafac0cf230 con 0x7fafc0104c00 2024-03-16T12:22:15.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.183+0000 7fafbea00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 0x7fafa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 -- 172.21.15.98:0/793020017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 msgr2=0x5642fc8d1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009fd0 tx=0x7fafb80118d0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 -- 172.21.15.98:0/793020017 shutdown_connections 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d7420 0x5642fc8d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d22b0 0x5642fc8d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 --2- 172.21.15.98:0/793020017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d18c0 0x5642fc8d1ce0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 -- 172.21.15.98:0/793020017 >> 172.21.15.98:0/793020017 conn(0x5642fc8cce30 msgr2=0x5642fc8cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 -- 172.21.15.98:0/793020017 shutdown_connections 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafc87ab740 1 -- 172.21.15.98:0/793020017 wait complete. 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.184+0000 7fafbea00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 0x7fafa40a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fafb002c180 tx=0x7fafb002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 -- start start 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 Processor -- start 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5642fc7f2110 con 0x5642fc8d18c0 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5642fc8324f0 con 0x5642fc8d22b0 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5642fc8da3d0 con 0x5642fc8d7420 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafc87ab740 10 client.0 _ll_register_callbacks cb 0x5642fc7dfb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafbea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.185+0000 7fafbe000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53030/0 (socket says 172.21.15.98:53030) 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8dbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53214/0 (socket says 172.21.15.98:53214) 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 -- 172.21.15.98:0/4166699842 learned_addr learned my addr 172.21.15.98:0/4166699842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 -- 172.21.15.98:0/4166699842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 msgr2=0x5642fc8e30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 -- 172.21.15.98:0/4166699842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 msgr2=0x5642fc8dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8dbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 -- 172.21.15.98:0/4166699842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642fc54a3f0 con 0x5642fc8d22b0 2024-03-16T12:22:15.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafbe000640 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fafac097be0 tx=0x7fafac097ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99301d48e6e73ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafa8c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac099950 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fafac0a3d20 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafb8004050 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.186+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fafac09bf10 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 10 client.25601 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5642fc6147d0 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb8004050 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fafac097040 con 0x5642fc8d22b0 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 10 client.25601 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 10 client.25601 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:22:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 20 client.25601 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc8e7960 0x5642fc8e9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 4 client.25601 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642fc843990 con 0x5642fc8e7960 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.187+0000 7fafbea00640 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc8e7960 0x5642fc8e9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.188+0000 7fafbea00640 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc8e7960 0x5642fc8e9d70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c405389f55918b4 server_cookie=ed5173f5c7e5eb07 in_seq=0 out_seq=0 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.188+0000 7fafa8c00640 10 client.25601 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.188+0000 7fafa8c00640 1 -- 172.21.15.98:0/4166699842 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642fc843990 con 0x5642fc8e7960 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.188+0000 7fafa8c00640 10 client.25601 handle_command_reply: tid=0 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.188+0000 7fafc87ab740 1 client.25601 shutdown 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 msgr2=0x5642fc8dc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fafac097be0 tx=0x7fafac097ae0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 shutdown_connections 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc8e7960 0x5642fc8e9d70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8dc0b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 --2- 172.21.15.98:0/4166699842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8dbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafa8c00640 0 client.25601 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 >> 172.21.15.98:0/4166699842 conn(0x5642fc8cce30 msgr2=0x5642fc8cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 shutdown_connections 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 1 -- 172.21.15.98:0/4166699842 wait complete. 2024-03-16T12:22:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.189+0000 7fafc87ab740 20 client.25601 trim_cache size 0 max 16384 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.203+0000 7fafc87ab740 1 Processor -- start 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 -- start start 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01aed90 con 0x5642fc8d22b0 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01ac660 con 0x5642fc8d7420 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafc87ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01aa4b0 con 0x5642fc8d18c0 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbe000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53040/0 (socket says 172.21.15.98:53040) 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbe000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53226/0 (socket says 172.21.15.98:53226) 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbf400640 1 -- 172.21.15.98:0/3620494398 learned_addr learned my addr 172.21.15.98:0/3620494398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.204+0000 7fafbea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57704/0 (socket says 172.21.15.98:57704) 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 -- 172.21.15.98:0/3620494398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 msgr2=0x5642fc8ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 -- 172.21.15.98:0/3620494398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 msgr2=0x5642fc8e30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 -- 172.21.15.98:0/3620494398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642fc8c7670 con 0x5642fc8d18c0 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbf400640 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbe000640 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafbea00640 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009230 tx=0x7fafb0041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519cd3c0d7b2531a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafab400640 1 -- 172.21.15.98:0/3620494398 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0004a90 con 0x5642fc8d18c0 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafab400640 1 -- 172.21.15.98:0/3620494398 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb0004c30 con 0x5642fc8d18c0 2024-03-16T12:22:15.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.205+0000 7fafab400640 1 -- 172.21.15.98:0/3620494398 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0005500 con 0x5642fc8d18c0 2024-03-16T12:22:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- 172.21.15.98:0/3620494398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 msgr2=0x5642fc8cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009230 tx=0x7fafb0041040 comp rx=0 tx=0).stop 2024-03-16T12:22:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- 172.21.15.98:0/3620494398 shutdown_connections 2024-03-16T12:22:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d7420 0x5642fc8ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d22b0 0x5642fc8e30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 --2- 172.21.15.98:0/3620494398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d18c0 0x5642fc8cc070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- 172.21.15.98:0/3620494398 >> 172.21.15.98:0/3620494398 conn(0x5642fc8cce30 msgr2=0x5642fc8cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- 172.21.15.98:0/3620494398 shutdown_connections 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- 172.21.15.98:0/3620494398 wait complete. 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 -- start start 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.206+0000 7fafc87ab740 1 Processor -- start 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01aab20 con 0x5642fc8d18c0 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01abf90 con 0x5642fc8d22b0 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafc01ab1f0 con 0x5642fc8d7420 2024-03-16T12:22:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafc87ab740 10 client.0 _ll_register_callbacks cb 0x7fafac00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbf400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53046/0 (socket says 172.21.15.98:53046) 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 -- 172.21.15.98:0/179704820 learned_addr learned my addr 172.21.15.98:0/179704820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 -- 172.21.15.98:0/179704820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 msgr2=0x5642fc8e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 -- 172.21.15.98:0/179704820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 msgr2=0x5642fc8e0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8e0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.207+0000 7fafbe000640 1 -- 172.21.15.98:0/179704820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642fc8e2600 con 0x5642fc8d22b0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafbe000640 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c6a0 tx=0x7fafac099820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b975223c8c63a669 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:15.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac099950 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fafac09a6d0 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafb0004c30 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fafac09b350 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 10 client.25607 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5642fc956840 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb0004c30 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fafac097040 con 0x5642fc8d22b0 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 10 client.25607 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 10 client.25607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 20 client.25607 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc957000 0x5642fc959420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 4 client.25607 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.208+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642fc83dce0 con 0x5642fc957000 2024-03-16T12:22:15.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafbea00640 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc957000 0x5642fc959420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafa8c00640 10 client.25607 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafbea00640 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc957000 0x5642fc959420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b4a2b7ff7b4c94c server_cookie=9fb695a7195e684a in_seq=0 out_seq=0 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafa8c00640 1 -- 172.21.15.98:0/179704820 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642fc83dce0 con 0x5642fc957000 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafa8c00640 10 client.25607 handle_command_reply: tid=0 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafc87ab740 1 client.25607 shutdown 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 msgr2=0x5642fc8e1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.209+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c6a0 tx=0x7fafac099820 comp rx=0 tx=0).stop 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 shutdown_connections 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x5642fc957000 0x5642fc959420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642fc8d7420 0x5642fc8e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5642fc8d22b0 0x5642fc8e1000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 --2- 172.21.15.98:0/179704820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5642fc8d18c0 0x5642fc8e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafa8c00640 0 client.25607 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 >> 172.21.15.98:0/179704820 conn(0x5642fc8cce30 msgr2=0x5642fc8cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 shutdown_connections 2024-03-16T12:22:15.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 1 -- 172.21.15.98:0/179704820 wait complete. 2024-03-16T12:22:15.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc87ab740 20 client.25607 trim_cache size 0 max 16384 2024-03-16T12:22:15.197 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 msgr2=0x7fafa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 0x7fafa40a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fafb002c180 tx=0x7fafb002cf90 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 msgr2=0x7fafc019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.210+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e9f0 tx=0x7fafac00edf0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 shutdown_connections 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fafa40a2880 0x7fafa40a4d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc01055c0 0x7fafc01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc0104c00 0x7fafc019a270 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 --2- 172.21.15.98:0/1236563917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafc0103a10 0x7fafc0199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 >> 172.21.15.98:0/1236563917 conn(0x7fafc00fefa0 msgr2=0x7fafc0107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 shutdown_connections 2024-03-16T12:22:15.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.211+0000 7fafc5a00640 1 -- 172.21.15.98:0/1236563917 wait complete. 2024-03-16T12:22:15.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:15 smithi098 ceph-mon[31633]: pgmap v427: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:15.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:15 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1637452966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:15.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T12:22:15.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:15 smithi173 ceph-mon[36632]: pgmap v427: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:15.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:15 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1637452966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 -- 172.21.15.98:0/2398474592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c074240 msgr2=0x7f5b5c0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 --2- 172.21.15.98:0/2398474592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c074240 0x7f5b5c0746c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009900 tx=0x7f5b4402bc40 comp rx=0 tx=0).stop 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 -- 172.21.15.98:0/2398474592 shutdown_connections 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 --2- 172.21.15.98:0/2398474592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c1094a0 0x7f5b5c10b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 --2- 172.21.15.98:0/2398474592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c074240 0x7f5b5c0746c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 --2- 172.21.15.98:0/2398474592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 0x7f5b5c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 -- 172.21.15.98:0/2398474592 >> 172.21.15.98:0/2398474592 conn(0x7f5b5c0fcb70 msgr2=0x7f5b5c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 -- 172.21.15.98:0/2398474592 shutdown_connections 2024-03-16T12:22:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.439+0000 7f5b62200640 1 -- 172.21.15.98:0/2398474592 wait complete. 2024-03-16T12:22:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.440+0000 7f5b62200640 1 Processor -- start 2024-03-16T12:22:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.440+0000 7f5b62200640 1 -- start start 2024-03-16T12:22:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 0x7f5b5c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c1094a0 0x7f5b5c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c10f490 con 0x7f5b5c075820 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c10f310 con 0x7f5b5c1094a0 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b62200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c10f610 con 0x7f5b5c074240 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57732/0 (socket says 172.21.15.98:57732) 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b53e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 0x7f5b5c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 -- 172.21.15.98:0/1139628849 learned_addr learned my addr 172.21.15.98:0/1139628849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b61800640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c1094a0 0x7f5b5c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 -- 172.21.15.98:0/1139628849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c1094a0 msgr2=0x7f5b5c1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c1094a0 0x7f5b5c1a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 -- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 msgr2=0x7f5b5c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 0x7f5b5c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.441+0000 7f5b60e00640 1 -- 172.21.15.98:0/1139628849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5c1a16c0 con 0x7f5b5c074240 2024-03-16T12:22:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b60e00640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00e9f0 tx=0x7f5b4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b5a000640 1 -- 172.21.15.98:0/1139628849 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c00cd80 con 0x7f5b5c074240 2024-03-16T12:22:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b5a000640 1 -- 172.21.15.98:0/1139628849 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b4c010910 con 0x7f5b5c074240 2024-03-16T12:22:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b5a000640 1 -- 172.21.15.98:0/1139628849 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c018990 con 0x7f5b5c074240 2024-03-16T12:22:15.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5c1a19b0 con 0x7f5b5c074240 2024-03-16T12:22:15.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.442+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5c1a1f20 con 0x7f5b5c074240 2024-03-16T12:22:15.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.445+0000 7f5b665d3740 1 Processor -- start 2024-03-16T12:22:15.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.445+0000 7f5b5a000640 1 -- 172.21.15.98:0/1139628849 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5b4c010ac0 con 0x7f5b5c074240 2024-03-16T12:22:15.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.445+0000 7f5b665d3740 1 -- start start 2024-03-16T12:22:15.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74a480 0x55e48d74ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d751a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e48d7113b0 con 0x55e48d749a90 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e48d6ae450 con 0x55e48d74a480 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e48d656cb0 con 0x55e48d74f5f0 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b61800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b60e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74a480 0x55e48d74ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b61800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53262/0 (socket says 172.21.15.98:53262) 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b61800640 1 -- 172.21.15.98:0/3768688953 learned_addr learned my addr 172.21.15.98:0/3768688953 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.446+0000 7f5b53e00640 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d751a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 -- 172.21.15.98:0/3768688953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 msgr2=0x55e48d751a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d751a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 -- 172.21.15.98:0/3768688953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74a480 msgr2=0x55e48d74ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74a480 0x55e48d74ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 -- 172.21.15.98:0/3768688953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e48d6781f0 con 0x55e48d749a90 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b53e00640 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d751a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b5a000640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 0x7f5b400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b61800640 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009fd0 tx=0x7f5b540118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285b2d15d26db568 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b5a000640 1 -- 172.21.15.98:0/1139628849 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b4c0cf070 con 0x7f5b5c074240 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b60e00640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 0x7f5b400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.447+0000 7f5b52000640 1 -- 172.21.15.98:0/3768688953 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54004d20 con 0x55e48d749a90 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b52000640 1 -- 172.21.15.98:0/3768688953 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b54004050 con 0x55e48d749a90 2024-03-16T12:22:15.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b52000640 1 -- 172.21.15.98:0/3768688953 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54005260 con 0x55e48d749a90 2024-03-16T12:22:15.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 -- 172.21.15.98:0/3768688953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 msgr2=0x55e48d749eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b60e00640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 0x7f5b400a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55e48d74a880 tx=0x7f5b4c0d2b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009fd0 tx=0x7f5b540118d0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 -- 172.21.15.98:0/3768688953 shutdown_connections 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d751a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74a480 0x55e48d74ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 --2- 172.21.15.98:0/3768688953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d749a90 0x55e48d749eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 -- 172.21.15.98:0/3768688953 >> 172.21.15.98:0/3768688953 conn(0x55e48d745000 msgr2=0x55e48d745400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 -- 172.21.15.98:0/3768688953 shutdown_connections 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.448+0000 7f5b665d3740 1 -- 172.21.15.98:0/3768688953 wait complete. 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 -- start start 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 Processor -- start 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 0x55e48d753de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d754320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74f5f0 0x55e48d75b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e48d66a110 con 0x55e48d74a480 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e48d6aa4f0 con 0x55e48d74f5f0 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.449+0000 7f5b665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e48d7525a0 con 0x55e48d749a90 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b665d3740 10 client.0 _ll_register_callbacks cb 0x55e48d657b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b61800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 0x55e48d753de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d754320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b61800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 0x55e48d753de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57746/0 (socket says 172.21.15.98:57746) 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b61800640 1 -- 172.21.15.98:0/1158749658 learned_addr learned my addr 172.21.15.98:0/1158749658 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b61800640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74f5f0 0x55e48d75b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 -- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 msgr2=0x55e48d753de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 0x55e48d753de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 -- 172.21.15.98:0/1158749658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74f5f0 msgr2=0x55e48d75b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74f5f0 0x55e48d75b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 -- 172.21.15.98:0/1158749658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e48d3c23f0 con 0x55e48d74a480 2024-03-16T12:22:15.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.450+0000 7f5b53e00640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d754320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009230 tx=0x7f5b44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773dc79815003c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44005d50 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b440046c0 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b54004050 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b440048c0 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b665d3740 10 client.15474 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e48d48c7d0 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.451+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b54004050 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b440035f0 con 0x55e48d74a480 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 10 client.15474 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 10 client.15474 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 20 client.15474 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d75fc50 0x55e48d762080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 4 client.15474 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:15.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e48d6bb990 con 0x55e48d75fc50 2024-03-16T12:22:15.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.452+0000 7f5b61800640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d75fc50 0x55e48d762080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b61800640 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d75fc50 0x55e48d762080 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56d128642022388e server_cookie=243e294bc91a3acc in_seq=0 out_seq=0 2024-03-16T12:22:15.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b4aa00640 10 client.15474 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b4aa00640 1 -- 172.21.15.98:0/1158749658 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e48d6bb990 con 0x55e48d75fc50 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b4aa00640 10 client.15474 handle_command_reply: tid=0 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b665d3740 1 client.15474 shutdown 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 msgr2=0x55e48d754320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.453+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d754320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009230 tx=0x7f5b44004270 comp rx=0 tx=0).stop 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 shutdown_connections 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d75fc50 0x55e48d762080 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d74f5f0 0x55e48d75b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d754320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b4aa00640 0 client.15474 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 --2- 172.21.15.98:0/1158749658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d749a90 0x55e48d753de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 >> 172.21.15.98:0/1158749658 conn(0x55e48d745000 msgr2=0x55e48d744220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 shutdown_connections 2024-03-16T12:22:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 1 -- 172.21.15.98:0/1158749658 wait complete. 2024-03-16T12:22:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.454+0000 7f5b665d3740 20 client.15474 trim_cache size 0 max 16384 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 Processor -- start 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 -- start start 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d743990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1aec50 con 0x55e48d74a480 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1ac520 con 0x55e48d749a90 2024-03-16T12:22:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1aa370 con 0x55e48d74f5f0 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b61800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b53e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b61800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53078/0 (socket says 172.21.15.98:53078) 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b61800640 1 -- 172.21.15.98:0/4125035294 learned_addr learned my addr 172.21.15.98:0/4125035294 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b53e00640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53278/0 (socket says 172.21.15.98:53278) 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.469+0000 7f5b61800640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d743990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 -- 172.21.15.98:0/4125035294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 msgr2=0x55e48d743990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d743990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 -- 172.21.15.98:0/4125035294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 msgr2=0x55e48d7453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 -- 172.21.15.98:0/4125035294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e48d744950 con 0x55e48d74a480 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b61800640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b53e00640 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b4402c720 tx=0x7f5b4402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c077d9b77710e9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b51600640 1 -- 172.21.15.98:0/4125035294 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44000e60 con 0x55e48d74a480 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b51600640 1 -- 172.21.15.98:0/4125035294 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44005850 con 0x55e48d74a480 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.470+0000 7f5b51600640 1 -- 172.21.15.98:0/4125035294 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440042e0 con 0x55e48d74a480 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- 172.21.15.98:0/4125035294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 msgr2=0x55e48d75b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b4402c720 tx=0x7f5b4402cd30 comp rx=0 tx=0).stop 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- 172.21.15.98:0/4125035294 shutdown_connections 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d743990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d75b2e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 --2- 172.21.15.98:0/4125035294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d7453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- 172.21.15.98:0/4125035294 >> 172.21.15.98:0/4125035294 conn(0x55e48d745000 msgr2=0x55e48d753db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- 172.21.15.98:0/4125035294 shutdown_connections 2024-03-16T12:22:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- 172.21.15.98:0/4125035294 wait complete. 2024-03-16T12:22:15.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.471+0000 7f5b665d3740 1 -- start start 2024-03-16T12:22:15.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 Processor -- start 2024-03-16T12:22:15.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d7593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d75a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1aa9e0 con 0x55e48d74a480 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1abe50 con 0x55e48d749a90 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c1ab0b0 con 0x55e48d74f5f0 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b665d3740 10 client.0 _ll_register_callbacks cb 0x7f5b44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b61800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b61800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53092/0 (socket says 172.21.15.98:53092) 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b61800640 1 -- 172.21.15.98:0/2264894703 learned_addr learned my addr 172.21.15.98:0/2264894703 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b53e00640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d7593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.472+0000 7f5b61800640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d75a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 -- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 msgr2=0x55e48d75a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d75a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 -- 172.21.15.98:0/2264894703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 msgr2=0x55e48d7593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d7593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 -- 172.21.15.98:0/2264894703 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e48d75a9f0 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b53e00640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d7593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b61800640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400a010 tx=0x7f5b5402f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d127b8ebcb389c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5402f430 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b5402f5d0 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.473+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b44005850 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5b5402f7d0 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 10 client.25625 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e48d7c64d0 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44005850 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5b54025040 con 0x55e48d749a90 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 10 client.25625 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 10 client.25625 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 20 client.25625 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d7c6ce0 0x55e48d7c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 4 client.25625 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e48d6b5ce0 con 0x55e48d7c6ce0 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.474+0000 7f5b53e00640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d7c6ce0 0x55e48d7c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b53e00640 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d7c6ce0 0x55e48d7c9100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d6690427b792120 server_cookie=d8288a1b3a672b1 in_seq=0 out_seq=0 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b4aa00640 10 client.25625 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b4aa00640 1 -- 172.21.15.98:0/2264894703 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e48d6b5ce0 con 0x55e48d7c6ce0 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b4aa00640 10 client.25625 handle_command_reply: tid=0 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 client.25625 shutdown 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 msgr2=0x55e48d758eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400a010 tx=0x7f5b5402f9c0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 shutdown_connections 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55e48d7c6ce0 0x55e48d7c9100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e48d74f5f0 0x55e48d75a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e48d74a480 0x55e48d7593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 --2- 172.21.15.98:0/2264894703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e48d749a90 0x55e48d758eb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b4aa00640 0 client.25625 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 >> 172.21.15.98:0/2264894703 conn(0x55e48d745000 msgr2=0x55e48d743990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 shutdown_connections 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.475+0000 7f5b665d3740 1 -- 172.21.15.98:0/2264894703 wait complete. 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b665d3740 20 client.25625 trim_cache size 0 max 16384 2024-03-16T12:22:15.461 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 msgr2=0x7f5b400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 0x7f5b400a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55e48d74a880 tx=0x7f5b4c0d2b40 comp rx=0 tx=0).stop 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 msgr2=0x7f5b5c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00e9f0 tx=0x7f5b4c00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 shutdown_connections 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5b400a2880 0x7f5b400a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b5c1094a0 0x7f5b5c1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b5c075820 0x7f5b5c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 --2- 172.21.15.98:0/1139628849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b5c074240 0x7f5b5c199aa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 >> 172.21.15.98:0/1139628849 conn(0x7f5b5c0fcb70 msgr2=0x7f5b5c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 shutdown_connections 2024-03-16T12:22:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.476+0000 7f5b62200640 1 -- 172.21.15.98:0/1139628849 wait complete. 2024-03-16T12:22:15.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T12:22:15.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:15 smithi153 ceph-mon[36901]: pgmap v427: 129 pgs: 129 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:15.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:15 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1637452966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.707+0000 7f0220800640 1 -- 172.21.15.98:0/1271192443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02181055c0 msgr2=0x7f021810be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.707+0000 7f0220800640 1 --2- 172.21.15.98:0/1271192443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02181055c0 0x7f021810be50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0214009ca0 tx=0x7f021402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 -- 172.21.15.98:0/1271192443 shutdown_connections 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 --2- 172.21.15.98:0/1271192443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02181055c0 0x7f021810be50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 --2- 172.21.15.98:0/1271192443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218104c00 0x7f0218105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 --2- 172.21.15.98:0/1271192443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0218103a10 0x7f0218103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 -- 172.21.15.98:0/1271192443 >> 172.21.15.98:0/1271192443 conn(0x7f02180fefa0 msgr2=0x7f02181013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 -- 172.21.15.98:0/1271192443 shutdown_connections 2024-03-16T12:22:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.708+0000 7f0220800640 1 -- 172.21.15.98:0/1271192443 wait complete. 2024-03-16T12:22:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 Processor -- start 2024-03-16T12:22:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 -- start start 2024-03-16T12:22:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218103a10 0x7f0218199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02181055c0 0x7f02181a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f021810f690 con 0x7f0218103a10 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f021810f510 con 0x7f02181055c0 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f0220800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f021810f810 con 0x7f0218104c00 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f021d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f021d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57780/0 (socket says 172.21.15.98:57780) 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f021d400640 1 -- 172.21.15.98:0/1607743737 learned_addr learned my addr 172.21.15.98:0/1607743737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f021de00640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218103a10 0x7f0218199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.709+0000 7f021e800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02181055c0 0x7f02181a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 -- 172.21.15.98:0/1607743737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02181055c0 msgr2=0x7f02181a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02181055c0 0x7f02181a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 -- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218103a10 msgr2=0x7f0218199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218103a10 0x7f0218199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 -- 172.21.15.98:0/1607743737 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02181a1920 con 0x7f0218104c00 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f021d400640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f020c00d950 tx=0x7f020c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f0212a00640 1 -- 172.21.15.98:0/1607743737 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c017030 con 0x7f0218104c00 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f0212a00640 1 -- 172.21.15.98:0/1607743737 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020c004020 con 0x7f0218104c00 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02181a1c10 con 0x7f0218104c00 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f0212a00640 1 -- 172.21.15.98:0/1607743737 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c0047a0 con 0x7f0218104c00 2024-03-16T12:22:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.710+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02181a2160 con 0x7f0218104c00 2024-03-16T12:22:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.713+0000 7f022357d740 1 Processor -- start 2024-03-16T12:22:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 -- start start 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f0212a00640 1 -- 172.21.15.98:0/1607743737 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f020c0041d0 con 0x7f0218104c00 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 0x55d8d0ebe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0e7e3b0 con 0x55d8d0eb68c0 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0e1b450 con 0x55d8d0ebc420 2024-03-16T12:22:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f022357d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0dc3cb0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f021de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f021de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57788/0 (socket says 172.21.15.98:57788) 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f021de00640 1 -- 172.21.15.98:0/1225659810 learned_addr learned my addr 172.21.15.98:0/1225659810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f021e800640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.714+0000 7f021d400640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 0x55d8d0ebe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 -- 172.21.15.98:0/1225659810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 msgr2=0x55d8d0ebe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 0x55d8d0ebe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 -- 172.21.15.98:0/1225659810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 msgr2=0x55d8d0eb6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 -- 172.21.15.98:0/1225659810 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8d0de51f0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021d400640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 0x55d8d0ebe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021e800640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f021de00640 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0204009c60 tx=0x7f0204011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4d8af43c865b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f020b000640 1 -- 172.21.15.98:0/1225659810 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204004220 con 0x55d8d0eb72b0 2024-03-16T12:22:15.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f020b000640 1 -- 172.21.15.98:0/1225659810 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02040043c0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f020b000640 1 -- 172.21.15.98:0/1225659810 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02040198a0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.715+0000 7f0212a00640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 -- 172.21.15.98:0/1225659810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 msgr2=0x55d8d0ebbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f0212a00640 1 -- 172.21.15.98:0/1607743737 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f020c0ce800 con 0x7f0218104c00 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0204009c60 tx=0x7f0204011810 comp rx=0 tx=0).stop 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f021e800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 -- 172.21.15.98:0/1225659810 shutdown_connections 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0ebc420 0x55d8d0ebe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ebbdb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 --2- 172.21.15.98:0/1225659810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 -- 172.21.15.98:0/1225659810 >> 172.21.15.98:0/1225659810 conn(0x55d8d0eb1e30 msgr2=0x55d8d0eb2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 -- 172.21.15.98:0/1225659810 shutdown_connections 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.716+0000 7f022357d740 1 -- 172.21.15.98:0/1225659810 wait complete. 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f021e800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 0x7f02000a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0214005e30 tx=0x7f021402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 -- start start 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 Processor -- start 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0dd7110 con 0x55d8d0eb72b0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0e174f0 con 0x55d8d0eb68c0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8d0ebf3d0 con 0x55d8d0ebc420 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f022357d740 10 client.0 _ll_register_callbacks cb 0x55d8d0dc4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f021de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f021de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53130/0 (socket says 172.21.15.98:53130) 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f021de00640 1 -- 172.21.15.98:0/2797040290 learned_addr learned my addr 172.21.15.98:0/2797040290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.717+0000 7f021d400640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 -- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 msgr2=0x55d8d0ec0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 -- 172.21.15.98:0/2797040290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb72b0 msgr2=0x55d8d0ec10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 -- 172.21.15.98:0/2797040290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8d0b2f3f0 con 0x55d8d0ebc420 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021de00640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f021e800640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec80e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0214009bf0 tx=0x7f021402c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d75ba14da40af7f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f01ff400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.718+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214051060 con 0x55d8d0ebc420 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02040043c0 con 0x55d8d0ebc420 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0214003b70 con 0x55d8d0ebc420 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0214058050 con 0x55d8d0ebc420 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 10 client.25624 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d8d0bf97d0 con 0x55d8d0ebc420 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02040043c0 con 0x55d8d0ebc420 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f021404c5f0 con 0x55d8d0ebc420 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 10 client.25624 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 10 client.25624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 20 client.25624 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0ecc9e0 0x55d8d0ecee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 4 client.25624 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.719+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8d0e28990 con 0x55d8d0ecc9e0 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.720+0000 7f021de00640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0ecc9e0 0x55d8d0ecee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.720+0000 7f021de00640 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0ecc9e0 0x55d8d0ecee20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94ad281c959691c9 server_cookie=aed4b906e42ed18c in_seq=0 out_seq=0 2024-03-16T12:22:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.720+0000 7f01ff400640 10 client.25624 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.721+0000 7f01ff400640 1 -- 172.21.15.98:0/2797040290 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8d0e28990 con 0x55d8d0ecc9e0 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.721+0000 7f01ff400640 10 client.25624 handle_command_reply: tid=0 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.721+0000 7f022357d740 1 client.25624 shutdown 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 msgr2=0x55d8d0ec80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec80e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0214009bf0 tx=0x7f021402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 shutdown_connections 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0ecc9e0 0x55d8d0ecee20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec80e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f01ff400640 0 client.25624 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 --2- 172.21.15.98:0/2797040290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 >> 172.21.15.98:0/2797040290 conn(0x55d8d0eb1e30 msgr2=0x55d8d0eb11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 shutdown_connections 2024-03-16T12:22:15.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 1 -- 172.21.15.98:0/2797040290 wait complete. 2024-03-16T12:22:15.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.722+0000 7f022357d740 20 client.25624 trim_cache size 0 max 16384 2024-03-16T12:22:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.736+0000 7f022357d740 1 Processor -- start 2024-03-16T12:22:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.736+0000 7f022357d740 1 -- start start 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 0x55d8d0ed36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02181aecf0 con 0x55d8d0ebc420 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02181ac5c0 con 0x55d8d0eb68c0 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f022357d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02181aa410 con 0x55d8d0eb72b0 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53136/0 (socket says 172.21.15.98:53136) 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021de00640 1 -- 172.21.15.98:0/919668504 learned_addr learned my addr 172.21.15.98:0/919668504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021e800640 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 0x55d8d0ed36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 -- 172.21.15.98:0/919668504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 msgr2=0x55d8d0eb1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 -- 172.21.15.98:0/919668504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 msgr2=0x55d8d0ed36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 0x55d8d0ed36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021d400640 1 -- 172.21.15.98:0/919668504 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8d0eadc10 con 0x55d8d0eb72b0 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.737+0000 7f021e800640 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 0x55d8d0ed36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f021d400640 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec80a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f020c09ad70 tx=0x7f020c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769b2ea831bd781d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f020a600640 1 -- 172.21.15.98:0/919668504 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c098f20 con 0x55d8d0eb72b0 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f020a600640 1 -- 172.21.15.98:0/919668504 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020c09b4e0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f020a600640 1 -- 172.21.15.98:0/919668504 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c09a490 con 0x55d8d0eb72b0 2024-03-16T12:22:15.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 -- 172.21.15.98:0/919668504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 msgr2=0x55d8d0ec80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec80a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f020c09ad70 tx=0x7f020c000bc0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 -- 172.21.15.98:0/919668504 shutdown_connections 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0ebc420 0x55d8d0ed36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec80a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 --2- 172.21.15.98:0/919668504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb68c0 0x55d8d0eb1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.738+0000 7f022357d740 1 -- 172.21.15.98:0/919668504 >> 172.21.15.98:0/919668504 conn(0x55d8d0eb1e30 msgr2=0x55d8d0eb0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 -- 172.21.15.98:0/919668504 shutdown_connections 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 -- 172.21.15.98:0/919668504 wait complete. 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 -- start start 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 Processor -- start 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.739+0000 7f022357d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f022357d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f022357d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02181aaa80 con 0x55d8d0eb68c0 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f022357d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02181abef0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f022357d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02181ab150 con 0x55d8d0ebc420 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f022357d740 10 client.0 _ll_register_callbacks cb 0x7f0204005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:53138/0 (socket says 172.21.15.98:53138) 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 -- 172.21.15.98:0/3268338877 learned_addr learned my addr 172.21.15.98:0/3268338877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021de00640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021e800640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 -- 172.21.15.98:0/3268338877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 msgr2=0x55d8d0ec6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 -- 172.21.15.98:0/3268338877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 msgr2=0x55d8d0ec5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 -- 172.21.15.98:0/3268338877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8d0ec7430 con 0x55d8d0eb72b0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021e800640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f021d400640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f020c005fd0 tx=0x7f020c098df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34f45fcbdba9ce96 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:15.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.740+0000 7f01ff400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c097530 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f020c09b4e0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f020c079bc0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f020c081050 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 10 client.25643 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d8d0f32e30 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020c09b4e0 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f020c09fe50 con 0x55d8d0eb72b0 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 10 client.25643 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 20 client.25643 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0f33750 0x55d8d0f35b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 4 client.25643 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:15.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.741+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8d0e22ce0 con 0x55d8d0f33750 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f021de00640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0f33750 0x55d8d0f35b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f021de00640 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0f33750 0x55d8d0f35b70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0e2ef05c2163c6 server_cookie=c8cbef8a668abdb in_seq=0 out_seq=0 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f01ff400640 10 client.25643 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f01ff400640 1 -- 172.21.15.98:0/3268338877 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8d0e22ce0 con 0x55d8d0f33750 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f01ff400640 10 client.25643 handle_command_reply: tid=0 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 client.25643 shutdown 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 msgr2=0x55d8d0ec5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f020c005fd0 tx=0x7f020c098df0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 shutdown_connections 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55d8d0f33750 0x55d8d0f35b70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8d0ebc420 0x55d8d0ec6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f01ff400640 0 client.25643 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d8d0eb72b0 0x55d8d0ec5e80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 --2- 172.21.15.98:0/3268338877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d8d0eb68c0 0x55d8d0ec5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.742+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 >> 172.21.15.98:0/3268338877 conn(0x55d8d0eb1e30 msgr2=0x55d8d0eb11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 shutdown_connections 2024-03-16T12:22:15.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f022357d740 1 -- 172.21.15.98:0/3268338877 wait complete. 2024-03-16T12:22:15.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f022357d740 20 client.25643 trim_cache size 0 max 16384 2024-03-16T12:22:15.729 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 msgr2=0x7f02000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 0x7f02000a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0214005e30 tx=0x7f021402e040 comp rx=0 tx=0).stop 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 msgr2=0x7f021819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f020c00d950 tx=0x7f020c00de10 comp rx=0 tx=0).stop 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 shutdown_connections 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f02000a2800 0x7f02000a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02181055c0 0x7f02181a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0218104c00 0x7f021819a250 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 --2- 172.21.15.98:0/1607743737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0218103a10 0x7f0218199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 >> 172.21.15.98:0/1607743737 conn(0x7f02180fefa0 msgr2=0x7f0218107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 shutdown_connections 2024-03-16T12:22:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:15.743+0000 7f0220800640 1 -- 172.21.15.98:0/1607743737 wait complete. 2024-03-16T12:22:16.742 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T12:22:16.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:22:16.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T12:22:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 -- 172.21.15.98:0/1181246278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281013e0 msgr2=0x7f5a28101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 --2- 172.21.15.98:0/1181246278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281013e0 0x7f5a28101860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009900 tx=0x7f5a2002bcd0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 -- 172.21.15.98:0/1181246278 shutdown_connections 2024-03-16T12:22:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 --2- 172.21.15.98:0/1181246278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281013e0 0x7f5a28101860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 --2- 172.21.15.98:0/1181246278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a28100a40 0x7f5a28100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 --2- 172.21.15.98:0/1181246278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a281000e0 0x7f5a28100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 -- 172.21.15.98:0/1181246278 >> 172.21.15.98:0/1181246278 conn(0x7f5a280fbc80 msgr2=0x7f5a280fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 -- 172.21.15.98:0/1181246278 shutdown_connections 2024-03-16T12:22:16.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.990+0000 7f5a30600640 1 -- 172.21.15.98:0/1181246278 wait complete. 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 Processor -- start 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 -- start start 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28100a40 0x7f5a2819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a281013e0 0x7f5a281a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810f6a0 con 0x7f5a28100a40 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810f520 con 0x7f5a281013e0 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.991+0000 7f5a30600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810f820 con 0x7f5a281000e0 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38734/0 (socket says 172.21.15.98:38734) 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 -- 172.21.15.98:0/4227671919 learned_addr learned my addr 172.21.15.98:0/4227671919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:16.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a281013e0 0x7f5a281a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28100a40 0x7f5a2819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 -- 172.21.15.98:0/4227671919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a281013e0 msgr2=0x7f5a281a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a281013e0 0x7f5a281a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 -- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28100a40 msgr2=0x7f5a2819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28100a40 0x7f5a2819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 -- 172.21.15.98:0/4227671919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a281a18f0 con 0x7f5a281000e0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00db80 tx=0x7f5a1c00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a26000640 1 -- 172.21.15.98:0/4227671919 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c017070 con 0x7f5a281000e0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.992+0000 7f5a26000640 1 -- 172.21.15.98:0/4227671919 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a1c003b70 con 0x7f5a281000e0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.993+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a281a1be0 con 0x7f5a281000e0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.993+0000 7f5a26000640 1 -- 172.21.15.98:0/4227671919 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c0106c0 con 0x7f5a281000e0 2024-03-16T12:22:16.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.993+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a281a2180 con 0x7f5a281000e0 2024-03-16T12:22:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.995+0000 7f5a26000640 1 -- 172.21.15.98:0/4227671919 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a1c004020 con 0x7f5a281000e0 2024-03-16T12:22:16.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 Processor -- start 2024-03-16T12:22:16.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 -- start start 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082e1d0 0x5588a0832cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a08357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588a0790f80 con 0x5588a082e1d0 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588a0739ad0 con 0x5588a082d7e0 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a3336a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588a078d020 con 0x5588a0833340 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.996+0000 7f5a2e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33736/0 (socket says 172.21.15.98:33736) 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 -- 172.21.15.98:0/2766521494 learned_addr learned my addr 172.21.15.98:0/2766521494 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2d200640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a08357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082e1d0 0x5588a0832cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 -- 172.21.15.98:0/2766521494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 msgr2=0x5588a08357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a08357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 -- 172.21.15.98:0/2766521494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082e1d0 msgr2=0x5588a0832cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082e1d0 0x5588a0832cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 -- 172.21.15.98:0/2766521494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588a07516f0 con 0x5588a082d7e0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2d200640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a08357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a26000640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 0x7f5a100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009900 tx=0x7f5a20009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1629450d864436c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a26000640 1 -- 172.21.15.98:0/4227671919 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a1c0ceb70 con 0x7f5a281000e0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.997+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 0x7f5a100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a1aa00640 1 -- 172.21.15.98:0/2766521494 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20043070 con 0x5588a082d7e0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a1aa00640 1 -- 172.21.15.98:0/2766521494 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a20004040 con 0x5588a082d7e0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a1aa00640 1 -- 172.21.15.98:0/2766521494 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20004f40 con 0x5588a082d7e0 2024-03-16T12:22:16.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a2dc00640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 0x7f5a100a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0975b0 tx=0x7f5a1c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 -- 172.21.15.98:0/2766521494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 msgr2=0x5588a082dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009900 tx=0x7f5a20009560 comp rx=0 tx=0).stop 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 -- 172.21.15.98:0/2766521494 shutdown_connections 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a08357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082e1d0 0x5588a0832cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2766521494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082d7e0 0x5588a082dc00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.998+0000 7f5a3336a740 1 -- 172.21.15.98:0/2766521494 >> 172.21.15.98:0/2766521494 conn(0x5588a07f5930 msgr2=0x5588a0829150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.999+0000 7f5a3336a740 1 -- 172.21.15.98:0/2766521494 shutdown_connections 2024-03-16T12:22:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.999+0000 7f5a3336a740 1 -- 172.21.15.98:0/2766521494 wait complete. 2024-03-16T12:22:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.999+0000 7f5a3336a740 1 -- start start 2024-03-16T12:22:16.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:16.999+0000 7f5a3336a740 1 Processor -- start 2024-03-16T12:22:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0837b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a0833340 0x5588a083f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5588a0836290 con 0x5588a082d7e0 2024-03-16T12:22:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588a074cf30 con 0x5588a0833340 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588a0836410 con 0x5588a082e1d0 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a3336a740 10 client.0 _ll_register_callbacks cb 0x5588a073abc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38758/0 (socket says 172.21.15.98:38758) 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 -- 172.21.15.98:0/1072184577 learned_addr learned my addr 172.21.15.98:0/1072184577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2e600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0837b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2e600640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a0833340 0x5588a083f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 -- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a0833340 msgr2=0x5588a083f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a0833340 0x5588a083f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 -- 172.21.15.98:0/1072184577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 msgr2=0x5588a0837b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0837b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2d200640 1 -- 172.21.15.98:0/1072184577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588a056d7e0 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.000+0000 7f5a2e600640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0837b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a2d200640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b9f0 tx=0x7f5a1400beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e81f44871ac91a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a14003dc0 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a20004040 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a14011440 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5a14011640 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a3336a740 10 client.25642 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5588a05cdf70 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a20004040 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.001+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5a1401b9a0 con 0x5588a082e1d0 2024-03-16T12:22:16.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 10 client.25642 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 20 client.25642 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a0843a50 0x5588a0845ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 4 client.25642 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588a079e4c0 con 0x5588a0843a50 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a2e600640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a0843a50 0x5588a0845ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a2e600640 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a0843a50 0x5588a0845ed0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7121b192f19d49c8 server_cookie=e117be7037541566 in_seq=0 out_seq=0 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.002+0000 7f5a0f400640 10 client.25642 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.003+0000 7f5a0f400640 1 -- 172.21.15.98:0/1072184577 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588a079e4c0 con 0x5588a0843a50 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.003+0000 7f5a0f400640 10 client.25642 handle_command_reply: tid=0 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.003+0000 7f5a3336a740 1 client.25642 shutdown 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.003+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 msgr2=0x5588a0838080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.003+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b9f0 tx=0x7f5a1400beb0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 shutdown_connections 2024-03-16T12:22:16.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a0833340 0x5588a083f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a0843a50 0x5588a0845ed0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082e1d0 0x5588a0838080 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 --2- 172.21.15.98:0/1072184577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0837b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 >> 172.21.15.98:0/1072184577 conn(0x5588a07f5930 msgr2=0x5588a0828350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a0f400640 0 client.25642 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 shutdown_connections 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 1 -- 172.21.15.98:0/1072184577 wait complete. 2024-03-16T12:22:16.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.004+0000 7f5a3336a740 20 client.25642 trim_cache size 0 max 16384 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.023+0000 7f5a3336a740 1 Processor -- start 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.023+0000 7f5a3336a740 1 -- start start 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0827ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a0828000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281aed30 con 0x5588a082d7e0 2024-03-16T12:22:17.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281ac600 con 0x5588a082e1d0 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a3336a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281aa450 con 0x5588a0833340 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33752/0 (socket says 172.21.15.98:33752) 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 -- 172.21.15.98:0/2149685014 learned_addr learned my addr 172.21.15.98:0/2149685014 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0827ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 -- 172.21.15.98:0/2149685014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 msgr2=0x5588a0828000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a0828000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a0828000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 -- 172.21.15.98:0/2149685014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 msgr2=0x5588a0827ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0827ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2d200640 1 -- 172.21.15.98:0/2149685014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588a082efe0 con 0x5588a082e1d0 2024-03-16T12:22:17.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.024+0000 7f5a2e600640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a0828000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.025+0000 7f5a2d200640 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006010 tx=0x7f5a14011570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc79fc1c276e177 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.025+0000 7f5a1a000640 1 -- 172.21.15.98:0/2149685014 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1402d070 con 0x5588a082e1d0 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.025+0000 7f5a1a000640 1 -- 172.21.15.98:0/2149685014 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a14011cc0 con 0x5588a082e1d0 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.025+0000 7f5a1a000640 1 -- 172.21.15.98:0/2149685014 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a14010c20 con 0x5588a082e1d0 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- 172.21.15.98:0/2149685014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 msgr2=0x5588a083f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006010 tx=0x7f5a14011570 comp rx=0 tx=0).stop 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- 172.21.15.98:0/2149685014 shutdown_connections 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a0833340 0x5588a0828000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083f080 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 --2- 172.21.15.98:0/2149685014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a082d7e0 0x5588a0827ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- 172.21.15.98:0/2149685014 >> 172.21.15.98:0/2149685014 conn(0x5588a07f5930 msgr2=0x5588a082f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- 172.21.15.98:0/2149685014 shutdown_connections 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- 172.21.15.98:0/2149685014 wait complete. 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 -- start start 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.026+0000 7f5a3336a740 1 Processor -- start 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082d7e0 0x5588a083c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a0833340 0x5588a083ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281aaac0 con 0x5588a0833340 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281abf30 con 0x5588a082e1d0 2024-03-16T12:22:17.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281ab190 con 0x5588a082d7e0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a3336a740 10 client.0 _ll_register_callbacks cb 0x7f5a14004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082d7e0 0x5588a083c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33756/0 (socket says 172.21.15.98:33756) 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 -- 172.21.15.98:0/761258360 learned_addr learned my addr 172.21.15.98:0/761258360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2e600640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a0833340 0x5588a083ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 -- 172.21.15.98:0/761258360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082d7e0 msgr2=0x5588a083c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082d7e0 0x5588a083c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 -- 172.21.15.98:0/761258360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a0833340 msgr2=0x5588a083ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a0833340 0x5588a083ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.027+0000 7f5a2d200640 1 -- 172.21.15.98:0/761258360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588a0465170 con 0x5588a082e1d0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a2d200640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5a14005fe0 tx=0x7f5a14011c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2af6d4baba6a2ec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a0f400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:17.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1402d040 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a14011cc0 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a14010960 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5a14010ba0 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a3336a740 10 client.25661 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5588a08a9dd0 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.028+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a14011cc0 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5a1401b940 con 0x5588a082e1d0 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 10 client.25661 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 10 client.25661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 20 client.25661 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a08aa350 0x5588a08ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 4 client.25661 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T12:22:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5588a0798810 con 0x5588a08aa350 2024-03-16T12:22:17.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.029+0000 7f5a2e600640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a08aa350 0x5588a08ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.030+0000 7f5a2e600640 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a08aa350 0x5588a08ac770 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97add5d1ccdf30a server_cookie=5b90432e4a159b76 in_seq=0 out_seq=0 2024-03-16T12:22:17.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.030+0000 7f5a0f400640 10 client.25661 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a0f400640 1 -- 172.21.15.98:0/761258360 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5588a0798810 con 0x5588a08aa350 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a0f400640 10 client.25661 handle_command_reply: tid=0 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 client.25661 shutdown 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 msgr2=0x5588a083ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5a14005fe0 tx=0x7f5a14011c00 comp rx=0 tx=0).stop 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 shutdown_connections 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5588a0833340 0x5588a083ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588a082e1d0 0x5588a083ce30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5588a08aa350 0x5588a08ac770 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 --2- 172.21.15.98:0/761258360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588a082d7e0 0x5588a083c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a0f400640 0 client.25661 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.031+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 >> 172.21.15.98:0/761258360 conn(0x5588a07f5930 msgr2=0x5588a082c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 shutdown_connections 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a3336a740 1 -- 172.21.15.98:0/761258360 wait complete. 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a3336a740 20 client.25661 trim_cache size 0 max 16384 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "252c46df-9f0e-4299-af26-a1ea43ada30d", 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-03-16T12:22:17.017 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 msgr2=0x7f5a100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 0x7f5a100a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0975b0 tx=0x7f5a1c000f40 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 msgr2=0x7f5a28199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00db80 tx=0x7f5a1c00b630 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 shutdown_connections 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f5a100a2880 0x7f5a100a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a281013e0 0x7f5a281a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a28100a40 0x7f5a2819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 --2- 172.21.15.98:0/4227671919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a281000e0 0x7f5a28199cb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.032+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 >> 172.21.15.98:0/4227671919 conn(0x7f5a280fbc80 msgr2=0x7f5a280fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.033+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 shutdown_connections 2024-03-16T12:22:17.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.033+0000 7f5a30600640 1 -- 172.21.15.98:0/4227671919 wait complete. 2024-03-16T12:22:17.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:22:17.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 -- 172.21.15.98:0/1461626703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281033a0 msgr2=0x7f1228109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 --2- 172.21.15.98:0/1461626703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281033a0 0x7f1228109c30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1220009ca0 tx=0x7f122002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 -- 172.21.15.98:0/1461626703 shutdown_connections 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 --2- 172.21.15.98:0/1461626703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281033a0 0x7f1228109c30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 --2- 172.21.15.98:0/1461626703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281029e0 0x7f1228102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 --2- 172.21.15.98:0/1461626703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281017f0 0x7f1228101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 -- 172.21.15.98:0/1461626703 >> 172.21.15.98:0/1461626703 conn(0x7f12280fcd60 msgr2=0x7f12280ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 -- 172.21.15.98:0/1461626703 shutdown_connections 2024-03-16T12:22:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.269+0000 7f122e000640 1 -- 172.21.15.98:0/1461626703 wait complete. 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 Processor -- start 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 -- start start 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281017f0 0x7f1228197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 0x7f122819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f122810d450 con 0x7f12281017f0 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f122810d2d0 con 0x7f12281029e0 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.270+0000 7f122e000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f122810d5d0 con 0x7f12281033a0 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281017f0 0x7f1228197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f121ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f121ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33774/0 (socket says 172.21.15.98:33774) 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f121ea00640 1 -- 172.21.15.98:0/2728817420 learned_addr learned my addr 172.21.15.98:0/2728817420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 0x7f122819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 -- 172.21.15.98:0/2728817420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 msgr2=0x7f1228198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 -- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281017f0 msgr2=0x7f1228197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281017f0 0x7f1228197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 -- 172.21.15.98:0/2728817420 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122819f6e0 con 0x7f12281033a0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f121ea00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1227e00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 0x7f122819f070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f122002c630 tx=0x7f1220032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.271+0000 7f1224c00640 1 -- 172.21.15.98:0/2728817420 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220041020 con 0x7f12281033a0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.272+0000 7f1224c00640 1 -- 172.21.15.98:0/2728817420 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1220034cd0 con 0x7f12281033a0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.272+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122819f970 con 0x7f12281033a0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.272+0000 7f1224c00640 1 -- 172.21.15.98:0/2728817420 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220039040 con 0x7f12281033a0 2024-03-16T12:22:17.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.272+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122819fe80 con 0x7f12281033a0 2024-03-16T12:22:17.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1224c00640 1 -- 172.21.15.98:0/2728817420 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1220044020 con 0x7f12281033a0 2024-03-16T12:22:17.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 Processor -- start 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 -- start start 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ae2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86ae7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ae9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f86a421f0 con 0x564f86ae7780 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f869eaa80 con 0x564f86ae1c20 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.275+0000 7f1230ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f86a3e290 con 0x564f86ae2610 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f121ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ae2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f121ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ae2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33784/0 (socket says 172.21.15.98:33784) 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f121ea00640 1 -- 172.21.15.98:0/923977394 learned_addr learned my addr 172.21.15.98:0/923977394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86ae7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f121ea00640 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ae9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 -- 172.21.15.98:0/923977394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 msgr2=0x564f86ae2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ae2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 -- 172.21.15.98:0/923977394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 msgr2=0x564f86ae9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ae9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 -- 172.21.15.98:0/923977394 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f868207d0 con 0x564f86ae2610 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1224c00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 0x7f120c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1227400640 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86ae7110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1218009bf0 tx=0x7f1218011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7aaf49572b11ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.276+0000 7f1224c00640 1 -- 172.21.15.98:0/2728817420 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12200f0b80 con 0x7f12281033a0 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1217e00640 1 -- 172.21.15.98:0/923977394 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218009da0 con 0x564f86ae2610 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1217e00640 1 -- 172.21.15.98:0/923977394 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1218005020 con 0x564f86ae2610 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1217e00640 1 -- 172.21.15.98:0/923977394 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218004b60 con 0x564f86ae2610 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f121ea00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 0x7f120c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 -- 172.21.15.98:0/923977394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 msgr2=0x564f86ae7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86ae7110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1218009bf0 tx=0x7f1218011810 comp rx=0 tx=0).stop 2024-03-16T12:22:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f121ea00640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 0x7f120c0a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x564f86ae2e70 tx=0x7f1210005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 -- 172.21.15.98:0/923977394 shutdown_connections 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ae9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86ae7110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 --2- 172.21.15.98:0/923977394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ae2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 -- 172.21.15.98:0/923977394 >> 172.21.15.98:0/923977394 conn(0x564f86ad8d80 msgr2=0x564f86ad9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 -- 172.21.15.98:0/923977394 shutdown_connections 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.277+0000 7f1230ded740 1 -- 172.21.15.98:0/923977394 wait complete. 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.278+0000 7f1230ded740 1 -- start start 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.278+0000 7f1230ded740 1 Processor -- start 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86aec170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f86aea730 con 0x564f86ae1c20 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f869fded0 con 0x564f86ae7780 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f86aea8b0 con 0x564f86ae2610 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1230ded740 10 client.0 _ll_register_callbacks cb 0x564f86a39220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1227400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86aec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38816/0 (socket says 172.21.15.98:38816) 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 -- 172.21.15.98:0/2587846452 learned_addr learned my addr 172.21.15.98:0/2587846452 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1227e00640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 -- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 msgr2=0x564f86af36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 -- 172.21.15.98:0/2587846452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 msgr2=0x564f86aec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86aec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 -- 172.21.15.98:0/2587846452 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f867563f0 con 0x564f86ae2610 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f121ea00640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1210004560 tx=0x7f1210004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=334df0d8b6c761a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.279+0000 7f1214c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210022430 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12100225d0 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1218005020 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f12100227d0 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1230ded740 10 client.25657 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f867160f0 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1218005020 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.280+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f121002a070 con 0x564f86ae2610 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 10 client.25657 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 10 client.25657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 20 client.25657 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86af8020 0x564f86afa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 4 client.25657 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:17.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f86a55be0 con 0x564f86af8020 2024-03-16T12:22:17.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1227400640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86af8020 0x564f86afa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1214c00640 10 client.25657 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.281+0000 7f1227400640 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86af8020 0x564f86afa4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8c0a41c9bcd724 server_cookie=8748a0624512d8a3 in_seq=0 out_seq=0 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.282+0000 7f1214c00640 1 -- 172.21.15.98:0/2587846452 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f86a55be0 con 0x564f86af8020 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.282+0000 7f1214c00640 10 client.25657 handle_command_reply: tid=0 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.282+0000 7f1230ded740 1 client.25657 shutdown 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 msgr2=0x564f86aec6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1210004560 tx=0x7f1210004730 comp rx=0 tx=0).stop 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 shutdown_connections 2024-03-16T12:22:17.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86af8020 0x564f86afa4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86aec6b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 --2- 172.21.15.98:0/2587846452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86aec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1214c00640 0 client.25657 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 >> 172.21.15.98:0/2587846452 conn(0x564f86ad8d80 msgr2=0x564f86ad7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 shutdown_connections 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 1 -- 172.21.15.98:0/2587846452 wait complete. 2024-03-16T12:22:17.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.283+0000 7f1230ded740 20 client.25657 trim_cache size 0 max 16384 2024-03-16T12:22:17.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.302+0000 7f1230ded740 1 Processor -- start 2024-03-16T12:22:17.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.302+0000 7f1230ded740 1 -- start start 2024-03-16T12:22:17.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ad9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ad7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12281aa9b0 con 0x564f86ae7780 2024-03-16T12:22:17.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a8280 con 0x564f86ae1c20 2024-03-16T12:22:17.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1230ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a60d0 con 0x564f86ae2610 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1227e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ad7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38818/0 (socket says 172.21.15.98:38818) 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 -- 172.21.15.98:0/1810489878 learned_addr learned my addr 172.21.15.98:0/1810489878 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f1227400640 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ad9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 -- 172.21.15.98:0/1810489878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 msgr2=0x564f86ad9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ad9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 -- 172.21.15.98:0/1810489878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 msgr2=0x564f86ad7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ad7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.303+0000 7f121ea00640 1 -- 172.21.15.98:0/1810489878 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f86aeca00 con 0x564f86ae2610 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.304+0000 7f121ea00640 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1210020c30 tx=0x7f121000fef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc559a523eb9a1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.304+0000 7f1217400640 1 -- 172.21.15.98:0/1810489878 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210003e50 con 0x564f86ae2610 2024-03-16T12:22:17.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.304+0000 7f1217400640 1 -- 172.21.15.98:0/1810489878 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1210022e50 con 0x564f86ae2610 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.304+0000 7f1217400640 1 -- 172.21.15.98:0/1810489878 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210020760 con 0x564f86ae2610 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- 172.21.15.98:0/1810489878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 msgr2=0x564f86af36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1210020c30 tx=0x7f121000fef0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- 172.21.15.98:0/1810489878 shutdown_connections 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae7780 0x564f86ad7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af36b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 --2- 172.21.15.98:0/1810489878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae1c20 0x564f86ad9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- 172.21.15.98:0/1810489878 >> 172.21.15.98:0/1810489878 conn(0x564f86ad8d80 msgr2=0x564f86ae3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- 172.21.15.98:0/1810489878 shutdown_connections 2024-03-16T12:22:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- 172.21.15.98:0/1810489878 wait complete. 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 -- start start 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.305+0000 7f1230ded740 1 Processor -- start 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a6740 con 0x564f86ae1c20 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a7bb0 con 0x564f86ae7780 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a6e10 con 0x564f86ae2610 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1230ded740 10 client.0 _ll_register_callbacks cb 0x7f1210009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1227400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1227400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41366/0 (socket says 172.21.15.98:41366) 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f121ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1227400640 1 -- 172.21.15.98:0/2552185778 learned_addr learned my addr 172.21.15.98:0/2552185778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.306+0000 7f1227e00640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 -- 172.21.15.98:0/2552185778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 msgr2=0x564f86af1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 -- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 msgr2=0x564f86af21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 -- 172.21.15.98:0/2552185778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f86af2790 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1227400640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1218005fd0 tx=0x7f12180097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8177b213af0a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1214c00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218005020 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1210022e50 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1218005460 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f121803a050 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1230ded740 10 client.15534 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.307+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f86b5e3e0 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1210022e50 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f12180035f0 con 0x564f86ae1c20 2024-03-16T12:22:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 10 client.15534 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 20 client.15534 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86b5ea90 0x564f86b60eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 4 client.15534 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f86a4f730 con 0x564f86b5ea90 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.308+0000 7f121ea00640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86b5ea90 0x564f86b60eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f121ea00640 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86b5ea90 0x564f86b60eb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411baf817cd14951 server_cookie=571b40d82010a662 in_seq=0 out_seq=0 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1214c00640 10 client.15534 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1214c00640 1 -- 172.21.15.98:0/2552185778 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f86a4f730 con 0x564f86b5ea90 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1214c00640 10 client.15534 handle_command_reply: tid=0 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 client.15534 shutdown 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 msgr2=0x564f86af0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1218005fd0 tx=0x7f12180097a0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 shutdown_connections 2024-03-16T12:22:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f86ae7780 0x564f86af21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f86b5ea90 0x564f86b60eb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f86ae2610 0x564f86af1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 --2- 172.21.15.98:0/2552185778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f86ae1c20 0x564f86af0c50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1214c00640 0 client.15534 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 >> 172.21.15.98:0/2552185778 conn(0x564f86ad8d80 msgr2=0x564f86ad7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 shutdown_connections 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.309+0000 7f1230ded740 1 -- 172.21.15.98:0/2552185778 wait complete. 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f1230ded740 20 client.15534 trim_cache size 0 max 16384 2024-03-16T12:22:17.294 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "252c46df-9f0e-4299-af26-a1ea43ada30d": { 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "252c46df-9f0e-4299-af26-a1ea43ada30d", 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:22:17.295 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 msgr2=0x7f120c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 0x7f120c0a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x564f86ae2e70 tx=0x7f1210005f90 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 msgr2=0x7f122819f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 0x7f122819f070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f122002c630 tx=0x7f1220032f20 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 shutdown_connections 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f120c0a2880 0x7f120c0a4d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12281033a0 0x7f122819f070 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12281029e0 0x7f1228198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 --2- 172.21.15.98:0/2728817420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12281017f0 0x7f1228197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 >> 172.21.15.98:0/2728817420 conn(0x7f12280fcd60 msgr2=0x7f1228105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.310+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 shutdown_connections 2024-03-16T12:22:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:17.311+0000 7f122e000640 1 -- 172.21.15.98:0/2728817420 wait complete. 2024-03-16T12:22:17.309 INFO:tasks.cephfs.filesystem:scrub status for tag:252c46df-9f0e-4299-af26-a1ea43ada30d - {'path': '/', 'tag': '252c46df-9f0e-4299-af26-a1ea43ada30d', 'options': 'recursive,force'} 2024-03-16T12:22:18.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:17 smithi098 ceph-mon[31633]: pgmap v428: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:18.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:17 smithi098 ceph-mon[31633]: scrub queued for path: / 2024-03-16T12:22:18.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:17 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:18.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:17 smithi098 ceph-mon[31633]: scrub summary: active paths [/] 2024-03-16T12:22:18.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:17 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:17 smithi153 ceph-mon[36901]: pgmap v428: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:17 smithi153 ceph-mon[36901]: scrub queued for path: / 2024-03-16T12:22:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:17 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:17 smithi153 ceph-mon[36901]: scrub summary: active paths [/] 2024-03-16T12:22:18.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:17 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:18.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:17 smithi173 ceph-mon[36632]: pgmap v428: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:18.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:17 smithi173 ceph-mon[36632]: scrub queued for path: / 2024-03-16T12:22:18.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:17 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:18.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:17 smithi173 ceph-mon[36632]: scrub summary: active paths [/] 2024-03-16T12:22:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:17 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:20.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:19 smithi098 ceph-mon[31633]: pgmap v429: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:20.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:19 smithi098 ceph-mon[31633]: scrub summary: idle 2024-03-16T12:22:20.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:19 smithi153 ceph-mon[36901]: pgmap v429: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:20.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:19 smithi153 ceph-mon[36901]: scrub summary: idle 2024-03-16T12:22:20.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:19 smithi173 ceph-mon[36632]: pgmap v429: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T12:22:20.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:19 smithi173 ceph-mon[36632]: scrub summary: idle 2024-03-16T12:22:22.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:21 smithi098 ceph-mon[31633]: pgmap v430: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T12:22:22.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:21 smithi153 ceph-mon[36901]: pgmap v430: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T12:22:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:21 smithi173 ceph-mon[36632]: pgmap v430: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T12:22:24.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:23 smithi098 ceph-mon[31633]: pgmap v431: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:24.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:23 smithi153 ceph-mon[36901]: pgmap v431: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:24.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:23 smithi173 ceph-mon[36632]: pgmap v431: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:26.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:25 smithi098 ceph-mon[31633]: pgmap v432: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:26.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:22:26.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:26.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:25 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:25 smithi153 ceph-mon[36901]: pgmap v432: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:22:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:26.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:25 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:26.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:25 smithi173 ceph-mon[36632]: pgmap v432: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:26.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:22:26.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:26.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:25 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:27.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:22:27.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:22:27.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:26 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:27.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:22:27.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:22:27.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:26 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:27.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:22:27.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:22:27.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:26 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:22:28.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:27 smithi173 ceph-mon[36632]: pgmap v433: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:28.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:27 smithi098 ceph-mon[31633]: pgmap v433: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:28.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:27 smithi153 ceph-mon[36901]: pgmap v433: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T12:22:30.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:29 smithi098 ceph-mon[31633]: pgmap v434: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:22:30.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:29 smithi153 ceph-mon[36901]: pgmap v434: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:22:30.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:29 smithi173 ceph-mon[36632]: pgmap v434: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:22:32.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:31 smithi098 ceph-mon[31633]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T12:22:32.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:31 smithi153 ceph-mon[36901]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T12:22:32.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:31 smithi173 ceph-mon[36632]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-16T12:22:34.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:33 smithi098 ceph-mon[31633]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:22:34.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:33 smithi153 ceph-mon[36901]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:22:34.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:33 smithi173 ceph-mon[36632]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:22:36.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:35 smithi098 ceph-mon[31633]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:36.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:35 smithi153 ceph-mon[36901]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:36.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:35 smithi173 ceph-mon[36632]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:38.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:37 smithi098 ceph-mon[31633]: pgmap v438: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:38.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:37 smithi153 ceph-mon[36901]: pgmap v438: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:38.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:37 smithi173 ceph-mon[36632]: pgmap v438: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:40.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:39 smithi098 ceph-mon[31633]: pgmap v439: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:40.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:39 smithi153 ceph-mon[36901]: pgmap v439: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:40.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:39 smithi173 ceph-mon[36632]: pgmap v439: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:22:42.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:41 smithi098 ceph-mon[31633]: pgmap v440: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:42.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:41 smithi153 ceph-mon[36901]: pgmap v440: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:42.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:41 smithi173 ceph-mon[36632]: pgmap v440: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:22:44.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:43 smithi098 ceph-mon[31633]: pgmap v441: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:44.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:43 smithi153 ceph-mon[36901]: pgmap v441: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:43 smithi173 ceph-mon[36632]: pgmap v441: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:46.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:45 smithi098 ceph-mon[31633]: pgmap v442: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:22:46.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:45 smithi153 ceph-mon[36901]: pgmap v442: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:22:46.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:45 smithi173 ceph-mon[36632]: pgmap v442: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:22:47.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:22:47.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:22:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 -- 172.21.15.98:0/4278188900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8075820 msgr2=0x7fb0e8073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 --2- 172.21.15.98:0/4278188900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8075820 0x7fb0e8073c70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009900 tx=0x7fb0d802bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 -- 172.21.15.98:0/4278188900 shutdown_connections 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 --2- 172.21.15.98:0/4278188900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e81095a0 0x7fb0e810ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 --2- 172.21.15.98:0/4278188900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e8074240 0x7fb0e80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 --2- 172.21.15.98:0/4278188900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8075820 0x7fb0e8073c70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.619+0000 7fb0ed600640 1 -- 172.21.15.98:0/4278188900 >> 172.21.15.98:0/4278188900 conn(0x7fb0e80fcc50 msgr2=0x7fb0e80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.620+0000 7fb0ed600640 1 -- 172.21.15.98:0/4278188900 shutdown_connections 2024-03-16T12:22:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.620+0000 7fb0ed600640 1 -- 172.21.15.98:0/4278188900 wait complete. 2024-03-16T12:22:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.621+0000 7fb0ed600640 1 Processor -- start 2024-03-16T12:22:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 -- start start 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e8075820 0x7fb0e819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 0x7fb0e81a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f570 con 0x7fb0e81095a0 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f3f0 con 0x7fb0e8074240 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0ed600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f6f0 con 0x7fb0e8075820 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.622+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54274/0 (socket says 172.21.15.98:54274) 2024-03-16T12:22:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 -- 172.21.15.98:0/2471479266 learned_addr learned my addr 172.21.15.98:0/2471479266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e8075820 0x7fb0e819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 0x7fb0e81a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 -- 172.21.15.98:0/2471479266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e8075820 msgr2=0x7fb0e819e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e8075820 0x7fb0e819e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 -- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 msgr2=0x7fb0e81a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 0x7fb0e81a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 -- 172.21.15.98:0/2471479266 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e806b330 con 0x7fb0e8074240 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 0x7fb0e81a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.623+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005ab0 tx=0x7fb0d802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.624+0000 7fb0e5600640 1 -- 172.21.15.98:0/2471479266 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8043020 con 0x7fb0e8074240 2024-03-16T12:22:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.624+0000 7fb0e5600640 1 -- 172.21.15.98:0/2471479266 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d80054e0 con 0x7fb0e8074240 2024-03-16T12:22:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.624+0000 7fb0e5600640 1 -- 172.21.15.98:0/2471479266 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8003d70 con 0x7fb0e8074240 2024-03-16T12:22:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.624+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e806b5c0 con 0x7fb0e8074240 2024-03-16T12:22:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.625+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e806bac0 con 0x7fb0e8074240 2024-03-16T12:22:47.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.629+0000 7fb0e5600640 1 -- 172.21.15.98:0/2471479266 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0d8046020 con 0x7fb0e8074240 2024-03-16T12:22:47.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.629+0000 7fb0f19a3740 1 Processor -- start 2024-03-16T12:22:47.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.629+0000 7fb0f19a3740 1 -- start start 2024-03-16T12:22:47.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 0x55950a5cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55950a52b1f0 con 0x55950a5c7400 2024-03-16T12:22:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55950a4d3a80 con 0x55950a5cc570 2024-03-16T12:22:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55950a527290 con 0x55950a5c6a10 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0ecc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0ecc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43734/0 (socket says 172.21.15.98:43734) 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0ecc00640 1 -- 172.21.15.98:0/380058216 learned_addr learned my addr 172.21.15.98:0/380058216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.630+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 0x55950a5cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0dfe00640 1 -- 172.21.15.98:0/380058216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 msgr2=0x55950a5c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0dfe00640 1 -- 172.21.15.98:0/380058216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 msgr2=0x55950a5cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0dfe00640 1 -- 172.21.15.98:0/380058216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55950a3097d0 con 0x55950a5cc570 2024-03-16T12:22:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.631+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.632+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 0x55950a5cea00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0005fd0 tx=0x7fb0d000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c148922981c8d33f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.632+0000 7fb0dd200640 1 -- 172.21.15.98:0/380058216 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0015070 con 0x55950a5cc570 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.632+0000 7fb0dd200640 1 -- 172.21.15.98:0/380058216 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d0005500 con 0x55950a5cc570 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.632+0000 7fb0dd200640 1 -- 172.21.15.98:0/380058216 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00116a0 con 0x55950a5cc570 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.632+0000 7fb0e5600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 0x7fb0cc0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.633+0000 7fb0e5600640 1 -- 172.21.15.98:0/2471479266 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb0d80f0c20 con 0x7fb0e8074240 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.633+0000 7fb0f19a3740 1 -- 172.21.15.98:0/380058216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 msgr2=0x55950a5cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.633+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 0x55950a5cea00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0005fd0 tx=0x7fb0d000f730 comp rx=0 tx=0).stop 2024-03-16T12:22:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.633+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 0x7fb0cc0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 -- 172.21.15.98:0/380058216 shutdown_connections 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5cc570 0x55950a5cea00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/380058216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5c6a10 0x55950a5c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 -- 172.21.15.98:0/380058216 >> 172.21.15.98:0/380058216 conn(0x55950a5c1f80 msgr2=0x55950a5c2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 0x7fb0cc0a4e10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55950a5c7c60 tx=0x7fb0e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 -- 172.21.15.98:0/380058216 shutdown_connections 2024-03-16T12:22:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.634+0000 7fb0f19a3740 1 -- 172.21.15.98:0/380058216 wait complete. 2024-03-16T12:22:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.635+0000 7fb0f19a3740 1 -- start start 2024-03-16T12:22:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.635+0000 7fb0f19a3740 1 Processor -- start 2024-03-16T12:22:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 0x55950a5d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c7400 0x55950a5d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55950a5cf520 con 0x55950a5c6a10 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55950a4e6ed0 con 0x55950a5c7400 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55950a5cf6a0 con 0x55950a5cc570 2024-03-16T12:22:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0f19a3740 10 client.0 _ll_register_callbacks cb 0x55950a522220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0ecc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 0x55950a5d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.636+0000 7fb0ecc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43742/0 (socket says 172.21.15.98:43742) 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0ecc00640 1 -- 172.21.15.98:0/720214839 learned_addr learned my addr 172.21.15.98:0/720214839 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c7400 0x55950a5d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0ecc00640 1 -- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c7400 msgr2=0x55950a5d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c7400 0x55950a5d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0ecc00640 1 -- 172.21.15.98:0/720214839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 msgr2=0x55950a5d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.637+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 0x55950a5d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.638+0000 7fb0ecc00640 1 -- 172.21.15.98:0/720214839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55950a23f3f0 con 0x55950a5cc570 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.638+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 0x55950a5d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.638+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0003e20 tx=0x7fb0e0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a07fb4f43e85dc5a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0d5600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0004450 con 0x55950a5cc570 2024-03-16T12:22:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb0e00045f0 con 0x55950a5cc570 2024-03-16T12:22:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d0005500 con 0x55950a5cc570 2024-03-16T12:22:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb0e0030050 con 0x55950a5cc570 2024-03-16T12:22:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0f19a3740 10 client.25675 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.639+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55950a1ff0f0 con 0x55950a5cc570 2024-03-16T12:22:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.640+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d0005500 con 0x55950a5cc570 2024-03-16T12:22:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.640+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb0e0027400 con 0x55950a5cc570 2024-03-16T12:22:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 10 client.25675 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 10 client.25675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 20 client.25675 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a5df370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 4 client.25675 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55950a53ebe0 con 0x55950a5dced0 2024-03-16T12:22:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a5df370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a5df370 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27596c82a260d0e2 server_cookie=4353e002e1e153e1 in_seq=0 out_seq=0 2024-03-16T12:22:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.641+0000 7fb0d5600640 10 client.25675 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:47.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.642+0000 7fb0d5600640 1 -- 172.21.15.98:0/720214839 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55950a53ebe0 con 0x55950a5dced0 2024-03-16T12:22:47.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.642+0000 7fb0d5600640 10 client.25675 handle_command_reply: tid=0 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.642+0000 7fb0f19a3740 1 client.25675 shutdown 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 msgr2=0x55950a5d8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0003e20 tx=0x7fb0e0004a40 comp rx=0 tx=0).stop 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 shutdown_connections 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a5df370 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0d5600640 0 client.25675 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d8530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c7400 0x55950a5d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/720214839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c6a10 0x55950a5d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 >> 172.21.15.98:0/720214839 conn(0x55950a5c1f80 msgr2=0x55950a5c1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 shutdown_connections 2024-03-16T12:22:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 1 -- 172.21.15.98:0/720214839 wait complete. 2024-03-16T12:22:47.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.643+0000 7fb0f19a3740 20 client.25675 trim_cache size 0 max 16384 2024-03-16T12:22:47.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 Processor -- start 2024-03-16T12:22:47.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 -- start start 2024-03-16T12:22:47.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81bb870 con 0x55950a5c7400 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81b9140 con 0x55950a5c6a10 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.679+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81b6f90 con 0x55950a5cc570 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54294/0 (socket says 172.21.15.98:54294) 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 -- 172.21.15.98:0/3638293373 learned_addr learned my addr 172.21.15.98:0/3638293373 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 -- 172.21.15.98:0/3638293373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 msgr2=0x55950a5c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 -- 172.21.15.98:0/3638293373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 msgr2=0x55950a5d84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 -- 172.21.15.98:0/3638293373 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55950a5dccb0 con 0x55950a5c6a10 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.680+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80bcf00 tx=0x7fb0d802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f76ce49e421944 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.681+0000 7fb0d7e00640 1 -- 172.21.15.98:0/3638293373 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004c80 con 0x55950a5c6a10 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.681+0000 7fb0d7e00640 1 -- 172.21.15.98:0/3638293373 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8005690 con 0x55950a5c6a10 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.681+0000 7fb0d7e00640 1 -- 172.21.15.98:0/3638293373 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004250 con 0x55950a5c6a10 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.682+0000 7fb0f19a3740 1 -- 172.21.15.98:0/3638293373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 msgr2=0x55950a5c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.682+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80bcf00 tx=0x7fb0d802bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 -- 172.21.15.98:0/3638293373 shutdown_connections 2024-03-16T12:22:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/3638293373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5c8900 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 -- 172.21.15.98:0/3638293373 >> 172.21.15.98:0/3638293373 conn(0x55950a5c1f80 msgr2=0x55950a5cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 -- 172.21.15.98:0/3638293373 shutdown_connections 2024-03-16T12:22:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 -- 172.21.15.98:0/3638293373 wait complete. 2024-03-16T12:22:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 -- start start 2024-03-16T12:22:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.683+0000 7fb0f19a3740 1 Processor -- start 2024-03-16T12:22:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81b7600 con 0x55950a5c7400 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81b8a70 con 0x55950a5c6a10 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81b7cd0 con 0x55950a5cc570 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0f19a3740 10 client.0 _ll_register_callbacks cb 0x7fb0e0009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54302/0 (socket says 172.21.15.98:54302) 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0e7e00640 1 -- 172.21.15.98:0/73144723 learned_addr learned my addr 172.21.15.98:0/73144723 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0dfe00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.684+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 -- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 msgr2=0x55950a5d5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 -- 172.21.15.98:0/73144723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 msgr2=0x55950a5d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 -- 172.21.15.98:0/73144723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55950a5d7660 con 0x55950a5cc570 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0ecc00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d7070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00279c0 tx=0x7fb0e0027690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b814a02e43f0e2f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0d5600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.685+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e002d040 con 0x55950a5cc570 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55950a538730 con 0x55950a5cc570 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb0e0027a40 con 0x55950a5cc570 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb0e0047050 con 0x55950a5cc570 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0f19a3740 10 client.25681 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55950a643270 con 0x55950a5cc570 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55950a538730 con 0x55950a5cc570 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb0e003c020 con 0x55950a5cc570 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0f19a3740 10 client.25681 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.686+0000 7fb0f19a3740 10 client.25681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0f19a3740 20 client.25681 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a645c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0f19a3740 4 client.25681 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55950a532a80 con 0x55950a5dced0 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a645c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0e7e00640 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a645c00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=114226c9d6f8e7bf server_cookie=1c762f6cb46edb5b in_seq=0 out_seq=0 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.687+0000 7fb0d5600640 10 client.25681 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0d5600640 1 -- 172.21.15.98:0/73144723 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55950a532a80 con 0x55950a5dced0 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0d5600640 10 client.25681 handle_command_reply: tid=0 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 client.25681 shutdown 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 msgr2=0x55950a5d7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d7070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00279c0 tx=0x7fb0e0027690 comp rx=0 tx=0).stop 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 shutdown_connections 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55950a5dced0 0x55950a645c00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55950a5cc570 0x55950a5d7070 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55950a5c7400 0x55950a5d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 --2- 172.21.15.98:0/73144723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55950a5c6a10 0x55950a5d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 >> 172.21.15.98:0/73144723 conn(0x55950a5c1f80 msgr2=0x55950a5cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0d5600640 0 client.25681 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 shutdown_connections 2024-03-16T12:22:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 1 -- 172.21.15.98:0/73144723 wait complete. 2024-03-16T12:22:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.688+0000 7fb0f19a3740 20 client.25681 trim_cache size 0 max 16384 2024-03-16T12:22:47.672 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:47.672 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-03-16T12:22:47.672 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-03-16T12:22:47.672 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 msgr2=0x7fb0cc0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 0x7fb0cc0a4e10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55950a5c7c60 tx=0x7fb0e0009500 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 msgr2=0x7fb0e819dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005ab0 tx=0x7fb0d802c6c0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 shutdown_connections 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb0cc0a2950 0x7fb0cc0a4e10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0e81095a0 0x7fb0e81a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e8075820 0x7fb0e819e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 --2- 172.21.15.98:0/2471479266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0e8074240 0x7fb0e819dfe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 >> 172.21.15.98:0/2471479266 conn(0x7fb0e80fcc50 msgr2=0x7fb0e8101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.689+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 shutdown_connections 2024-03-16T12:22:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.690+0000 7fb0ed600640 1 -- 172.21.15.98:0/2471479266 wait complete. 2024-03-16T12:22:47.686 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T12:22:47.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:22:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 -- 172.21.15.98:0/3088655573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62180ffe60 msgr2=0x7f6218100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 --2- 172.21.15.98:0/3088655573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62180ffe60 0x7f6218100280 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6208009900 tx=0x7f620802bc40 comp rx=0 tx=0).stop 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 -- 172.21.15.98:0/3088655573 shutdown_connections 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 --2- 172.21.15.98:0/3088655573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 --2- 172.21.15.98:0/3088655573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62181007c0 0x7f6218100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 --2- 172.21.15.98:0/3088655573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62180ffe60 0x7f6218100280 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 -- 172.21.15.98:0/3088655573 >> 172.21.15.98:0/3088655573 conn(0x7f62180fba60 msgr2=0x7f62180fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 -- 172.21.15.98:0/3088655573 shutdown_connections 2024-03-16T12:22:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.886+0000 7f621ca00640 1 -- 172.21.15.98:0/3088655573 wait complete. 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.887+0000 7f621ca00640 1 Processor -- start 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 -- start start 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62181007c0 0x7f6218199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f621810f4a0 con 0x7f6218101160 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f621810f320 con 0x7f62181007c0 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f621ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f621810f620 con 0x7f62180ffe60 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.888+0000 7f6217400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43774/0 (socket says 172.21.15.98:43774) 2024-03-16T12:22:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 -- 172.21.15.98:0/2928585052 learned_addr learned my addr 172.21.15.98:0/2928585052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6216a00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62181007c0 0x7f6218199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217e00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 -- 172.21.15.98:0/2928585052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62181007c0 msgr2=0x7f6218199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62181007c0 0x7f6218199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 -- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 msgr2=0x7f62181a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 -- 172.21.15.98:0/2928585052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62181a16a0 con 0x7f62180ffe60 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217e00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.889+0000 7f6217400640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f620802c8f0 tx=0x7f6208005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.890+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208041070 con 0x7f62180ffe60 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.890+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62080044d0 con 0x7f62180ffe60 2024-03-16T12:22:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.890+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208034440 con 0x7f62180ffe60 2024-03-16T12:22:47.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.890+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62181a1930 con 0x7f62180ffe60 2024-03-16T12:22:47.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.890+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62181a1dd0 con 0x7f62180ffe60 2024-03-16T12:22:47.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.891+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f621810ac80 con 0x7f62180ffe60 2024-03-16T12:22:47.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.895+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6208044070 con 0x7f62180ffe60 2024-03-16T12:22:47.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.897+0000 7f620fe00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:47.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.897+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62080f09b0 con 0x7f62180ffe60 2024-03-16T12:22:47.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.898+0000 7f6216a00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:47.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.898+0000 7f6216a00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 0x7f61fc0a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f621819af90 tx=0x7f6200005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:47.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:47.900+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62080bd690 con 0x7f62180ffe60 2024-03-16T12:22:47.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:47 smithi098 ceph-mon[31633]: pgmap v443: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:22:48.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:47 smithi153 ceph-mon[36901]: pgmap v443: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:22:48.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:47 smithi173 ceph-mon[36632]: pgmap v443: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:22:48.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.360+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f621802d2d0 con 0x7f62180ffe60 2024-03-16T12:22:48.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.361+0000 7f620fe00640 1 -- 172.21.15.98:0/2928585052 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7f62080bcf90 con 0x7f62180ffe60 2024-03-16T12:22:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:22:48.342 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:22:48.343 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:22:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.364+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 msgr2=0x7f61fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.364+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 0x7f61fc0a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f621819af90 tx=0x7f6200005f90 comp rx=0 tx=0).stop 2024-03-16T12:22:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.364+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 msgr2=0x7f6218199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.364+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f620802c8f0 tx=0x7f6208005d50 comp rx=0 tx=0).stop 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 shutdown_connections 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6218101160 0x7f62181a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62181007c0 0x7f6218199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 --2- 172.21.15.98:0/2928585052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62180ffe60 0x7f6218199a80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 >> 172.21.15.98:0/2928585052 conn(0x7f62180fba60 msgr2=0x7f62180fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 shutdown_connections 2024-03-16T12:22:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.365+0000 7f621ca00640 1 -- 172.21.15.98:0/2928585052 wait complete. 2024-03-16T12:22:48.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 -- 172.21.15.98:0/273711544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 msgr2=0x7fc210077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 --2- 172.21.15.98:0/273711544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc210077840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009900 tx=0x7fc1fc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 -- 172.21.15.98:0/273711544 shutdown_connections 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 --2- 172.21.15.98:0/273711544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc210078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 --2- 172.21.15.98:0/273711544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc210077840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 --2- 172.21.15.98:0/273711544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc2100765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 -- 172.21.15.98:0/273711544 >> 172.21.15.98:0/273711544 conn(0x7fc2100fef90 msgr2=0x7fc2101013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 -- 172.21.15.98:0/273711544 shutdown_connections 2024-03-16T12:22:48.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.684+0000 7fc216600640 1 -- 172.21.15.98:0/273711544 wait complete. 2024-03-16T12:22:48.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.685+0000 7fc216600640 1 Processor -- start 2024-03-16T12:22:48.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 -- start start 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc21019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc2101a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc210113ad0 con 0x7fc2100761a0 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc210113950 con 0x7fc210077d80 2024-03-16T12:22:48.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.686+0000 7fc216600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc210113c50 con 0x7fc2100773e0 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc2101a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47198/0 (socket says 172.21.15.98:47198) 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 -- 172.21.15.98:0/3902035831 learned_addr learned my addr 172.21.15.98:0/3902035831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc2101a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54338/0 (socket says 172.21.15.98:54338) 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20ea00640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc21019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 -- 172.21.15.98:0/3902035831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 msgr2=0x7fc21019e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc21019e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 -- 172.21.15.98:0/3902035831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 msgr2=0x7fc2101a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc2101a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.687+0000 7fc20f400640 1 -- 172.21.15.98:0/3902035831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2101a5cb0 con 0x7fc2100761a0 2024-03-16T12:22:48.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.688+0000 7fc20f400640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc20000d950 tx=0x7fc20000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.688+0000 7fc207e00640 1 -- 172.21.15.98:0/3902035831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200017070 con 0x7fc2100761a0 2024-03-16T12:22:48.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.688+0000 7fc207e00640 1 -- 172.21.15.98:0/3902035831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc200009c30 con 0x7fc2100761a0 2024-03-16T12:22:48.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.688+0000 7fc207e00640 1 -- 172.21.15.98:0/3902035831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000f6b0 con 0x7fc2100761a0 2024-03-16T12:22:48.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.689+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2101a5fa0 con 0x7fc2100761a0 2024-03-16T12:22:48.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.689+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101a63b0 con 0x7fc2100761a0 2024-03-16T12:22:48.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 Processor -- start 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 -- start start 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 0x560bc6b85ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b8adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 0x560bc6b8d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bc6b4d3b0 con 0x560bc6b862b0 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560bc6aea450 con 0x560bc6b858c0 2024-03-16T12:22:48.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.694+0000 7fc2193a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560bc6a92cb0 con 0x560bc6b8b420 2024-03-16T12:22:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc207e00640 1 -- 172.21.15.98:0/3902035831 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc2000109d0 con 0x7fc2100761a0 2024-03-16T12:22:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 0x560bc6b8d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 0x560bc6b8d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43800/0 (socket says 172.21.15.98:43800) 2024-03-16T12:22:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20ea00640 1 -- 172.21.15.98:0/1414631106 learned_addr learned my addr 172.21.15.98:0/1414631106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 0x560bc6b85ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 -- 172.21.15.98:0/1414631106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 msgr2=0x560bc6b8d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 0x560bc6b8d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 -- 172.21.15.98:0/1414631106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 msgr2=0x560bc6b8adb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b8adb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.695+0000 7fc20fe00640 1 -- 172.21.15.98:0/1414631106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc6ab41f0 con 0x560bc6b858c0 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.696+0000 7fc20fe00640 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 0x560bc6b85ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc208009f30 tx=0x7fc208010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b49b216e96f5fbe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc204c00640 1 -- 172.21.15.98:0/1414631106 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208016070 con 0x560bc6b858c0 2024-03-16T12:22:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc204c00640 1 -- 172.21.15.98:0/1414631106 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc208004d20 con 0x560bc6b858c0 2024-03-16T12:22:48.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc204c00640 1 -- 172.21.15.98:0/1414631106 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208012870 con 0x560bc6b858c0 2024-03-16T12:22:48.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc207e00640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 0x7fc1f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc207e00640 1 -- 172.21.15.98:0/3902035831 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc2000ce8f0 con 0x7fc2100761a0 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.697+0000 7fc20f400640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 0x7fc1f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 -- 172.21.15.98:0/1414631106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 msgr2=0x560bc6b85ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 0x560bc6b85ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc208009f30 tx=0x7fc208010800 comp rx=0 tx=0).stop 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 -- 172.21.15.98:0/1414631106 shutdown_connections 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b8b420 0x560bc6b8d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b8adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1414631106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b858c0 0x560bc6b85ce0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 -- 172.21.15.98:0/1414631106 >> 172.21.15.98:0/1414631106 conn(0x560bc6b80e30 msgr2=0x560bc6b81230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 -- 172.21.15.98:0/1414631106 shutdown_connections 2024-03-16T12:22:48.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc2193a6740 1 -- 172.21.15.98:0/1414631106 wait complete. 2024-03-16T12:22:48.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.698+0000 7fc20f400640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 0x7fc1f40a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc2000a31a0 tx=0x7fc200005880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.699+0000 7fc2193a6740 1 -- start start 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.699+0000 7fc2193a6740 1 Processor -- start 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 0x560bc6b8fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b862b0 0x560bc6b90110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b97150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bc6aa6110 con 0x560bc6b858c0 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560bc6ae64f0 con 0x560bc6b8b420 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560bc6b8e3d0 con 0x560bc6b862b0 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc2193a6740 10 client.0 _ll_register_callbacks cb 0x560bc6a93b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 0x560bc6b8fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 0x560bc6b8fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47214/0 (socket says 172.21.15.98:47214) 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc20fe00640 1 -- 172.21.15.98:0/172213123 learned_addr learned my addr 172.21.15.98:0/172213123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc20ea00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b862b0 0x560bc6b90110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.700+0000 7fc20fe00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b97150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 -- 172.21.15.98:0/172213123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b862b0 msgr2=0x560bc6b90110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b862b0 0x560bc6b90110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 -- 172.21.15.98:0/172213123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 msgr2=0x560bc6b8fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 0x560bc6b8fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 -- 172.21.15.98:0/172213123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc67fe3f0 con 0x560bc6b8b420 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc20fe00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b97150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098b0 tx=0x7fc208009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69e83db9f4b9baa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.701+0000 7fc1f9600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208016070 con 0x560bc6b8b420 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc208022030 con 0x560bc6b8b420 2024-03-16T12:22:48.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc208004d20 con 0x560bc6b8b420 2024-03-16T12:22:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc208005620 con 0x560bc6b8b420 2024-03-16T12:22:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc2193a6740 10 client.25715 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.702+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bc68c87d0 con 0x560bc6b8b420 2024-03-16T12:22:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc208004d20 con 0x560bc6b8b420 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc20800f040 con 0x560bc6b8b420 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 10 client.25715 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 10 client.25715 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 20 client.25715 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6b9b6f0 0x560bc6b9db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 4 client.25715 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.703+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bc6af7990 con 0x560bc6b9b6f0 2024-03-16T12:22:48.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.704+0000 7fc20ea00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6b9b6f0 0x560bc6b9db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.704+0000 7fc20ea00640 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6b9b6f0 0x560bc6b9db00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9826747b28bb45d0 server_cookie=a43ee0e52744e6dc in_seq=0 out_seq=0 2024-03-16T12:22:48.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.705+0000 7fc1f9600640 10 client.25715 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:48.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.705+0000 7fc1f9600640 1 -- 172.21.15.98:0/172213123 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bc6af7990 con 0x560bc6b9b6f0 2024-03-16T12:22:48.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.705+0000 7fc1f9600640 10 client.25715 handle_command_reply: tid=0 2024-03-16T12:22:48.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 client.25715 shutdown 2024-03-16T12:22:48.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 msgr2=0x560bc6b97150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b97150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098b0 tx=0x7fc208009600 comp rx=0 tx=0).stop 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 shutdown_connections 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6b9b6f0 0x560bc6b9db00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b97150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b862b0 0x560bc6b90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 --2- 172.21.15.98:0/172213123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b858c0 0x560bc6b8fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.706+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 >> 172.21.15.98:0/172213123 conn(0x560bc6b80e30 msgr2=0x560bc6b801f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.707+0000 7fc1f9600640 0 client.25715 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.707+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 shutdown_connections 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.707+0000 7fc2193a6740 1 -- 172.21.15.98:0/172213123 wait complete. 2024-03-16T12:22:48.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.707+0000 7fc2193a6740 20 client.25715 trim_cache size 0 max 16384 2024-03-16T12:22:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 Processor -- start 2024-03-16T12:22:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 -- start start 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b7f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b97110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b7fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101b3290 con 0x560bc6b862b0 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101b0b60 con 0x560bc6b8b420 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.735+0000 7fc2193a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101ae9b0 con 0x560bc6b858c0 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b7f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b97110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b7f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43818/0 (socket says 172.21.15.98:43818) 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20fe00640 1 -- 172.21.15.98:0/1577487082 learned_addr learned my addr 172.21.15.98:0/1577487082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20fe00640 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b7fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 -- 172.21.15.98:0/1577487082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 msgr2=0x560bc6b7f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b7f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 -- 172.21.15.98:0/1577487082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 msgr2=0x560bc6b7fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b7fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.736+0000 7fc20ea00640 1 -- 172.21.15.98:0/1577487082 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc6b7cc10 con 0x560bc6b862b0 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.737+0000 7fc20ea00640 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b97110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005ab0 tx=0x7fc1fc02cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d89b88c0341699b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.737+0000 7fc1fbe00640 1 -- 172.21.15.98:0/1577487082 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc02cd40 con 0x560bc6b862b0 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.737+0000 7fc1fbe00640 1 -- 172.21.15.98:0/1577487082 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1fc04dba0 con 0x560bc6b862b0 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.737+0000 7fc1fbe00640 1 -- 172.21.15.98:0/1577487082 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc004eb0 con 0x560bc6b862b0 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.738+0000 7fc2193a6740 1 -- 172.21.15.98:0/1577487082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 msgr2=0x560bc6b97110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.738+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b97110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005ab0 tx=0x7fc1fc02cc10 comp rx=0 tx=0).stop 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 -- 172.21.15.98:0/1577487082 shutdown_connections 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b8b420 0x560bc6b7fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b862b0 0x560bc6b97110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 --2- 172.21.15.98:0/1577487082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 -- 172.21.15.98:0/1577487082 >> 172.21.15.98:0/1577487082 conn(0x560bc6b80e30 msgr2=0x560bc6b87710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 -- 172.21.15.98:0/1577487082 shutdown_connections 2024-03-16T12:22:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.739+0000 7fc2193a6740 1 -- 172.21.15.98:0/1577487082 wait complete. 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 -- start start 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 Processor -- start 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b862b0 0x560bc6b94880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b8b420 0x560bc6b95840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101af020 con 0x560bc6b8b420 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101b0490 con 0x560bc6b862b0 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2101af6f0 con 0x560bc6b858c0 2024-03-16T12:22:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.740+0000 7fc2193a6740 10 client.0 _ll_register_callbacks cb 0x7fc1fc005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43832/0 (socket says 172.21.15.98:43832) 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 -- 172.21.15.98:0/2356879373 learned_addr learned my addr 172.21.15.98:0/2356879373 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b8b420 0x560bc6b95840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20ea00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b862b0 0x560bc6b94880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 -- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b862b0 msgr2=0x560bc6b94880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b862b0 0x560bc6b94880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 -- 172.21.15.98:0/2356879373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b8b420 msgr2=0x560bc6b95840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b8b420 0x560bc6b95840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.741+0000 7fc20fe00640 1 -- 172.21.15.98:0/2356879373 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc6b95e30 con 0x560bc6b858c0 2024-03-16T12:22:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc20fe00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc208033700 tx=0x7fc208005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7c3bbf5a800684 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc1f9600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208016040 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc208022040 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1fc04dba0 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc208047050 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc2193a6740 10 client.25705 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.742+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bc6b97130 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1fc04dba0 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc20803f070 con 0x560bc6b858c0 2024-03-16T12:22:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 10 client.25705 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 10 client.25705 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 20 client.25705 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6c01e30 0x560bc6c04250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 4 client.25705 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.743+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bc6af1ce0 con 0x560bc6c01e30 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc20ea00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6c01e30 0x560bc6c04250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc20ea00640 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6c01e30 0x560bc6c04250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e72168fae879d1a server_cookie=1b1309e0918f19d1 in_seq=0 out_seq=0 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc1f9600640 10 client.25705 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc1f9600640 1 -- 172.21.15.98:0/2356879373 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bc6af1ce0 con 0x560bc6c01e30 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc1f9600640 10 client.25705 handle_command_reply: tid=0 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc2193a6740 1 client.25705 shutdown 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 msgr2=0x560bc6b94340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.744+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc208033700 tx=0x7fc208005370 comp rx=0 tx=0).stop 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 shutdown_connections 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560bc6c01e30 0x560bc6c04250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc6b8b420 0x560bc6b95840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560bc6b862b0 0x560bc6b94880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 --2- 172.21.15.98:0/2356879373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560bc6b858c0 0x560bc6b94340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 >> 172.21.15.98:0/2356879373 conn(0x560bc6b80e30 msgr2=0x560bc6b84090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc1f9600640 0 client.25705 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 shutdown_connections 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 1 -- 172.21.15.98:0/2356879373 wait complete. 2024-03-16T12:22:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.745+0000 7fc2193a6740 20 client.25705 trim_cache size 0 max 16384 2024-03-16T12:22:48.730 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 msgr2=0x7fc1f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 0x7fc1f40a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc2000a31a0 tx=0x7fc200005880 comp rx=0 tx=0).stop 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 msgr2=0x7fc21019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc20000d950 tx=0x7fc20000de10 comp rx=0 tx=0).stop 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 shutdown_connections 2024-03-16T12:22:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1f40a2880 0x7fc1f40a4d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc210077d80 0x7fc2101a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2100773e0 0x7fc21019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 --2- 172.21.15.98:0/3902035831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2100761a0 0x7fc21019e090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 >> 172.21.15.98:0/3902035831 conn(0x7fc2100fef90 msgr2=0x7fc210100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.746+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 shutdown_connections 2024-03-16T12:22:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.747+0000 7fc216600640 1 -- 172.21.15.98:0/3902035831 wait complete. 2024-03-16T12:22:48.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.964+0000 7f3e1e600640 1 -- 172.21.15.98:0/4116536056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e18077d80 msgr2=0x7f3e18078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.964+0000 7f3e1e600640 1 --2- 172.21.15.98:0/4116536056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e18077d80 0x7f3e18078620 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009ca0 tx=0x7f3e1002c100 comp rx=0 tx=0).stop 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 -- 172.21.15.98:0/4116536056 shutdown_connections 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 --2- 172.21.15.98:0/4116536056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e18077d80 0x7f3e18078620 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 --2- 172.21.15.98:0/4116536056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e180773e0 0x7f3e18077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 --2- 172.21.15.98:0/4116536056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e180765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 -- 172.21.15.98:0/4116536056 >> 172.21.15.98:0/4116536056 conn(0x7f3e180fef90 msgr2=0x7f3e181013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 -- 172.21.15.98:0/4116536056 shutdown_connections 2024-03-16T12:22:48.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.965+0000 7f3e1e600640 1 -- 172.21.15.98:0/4116536056 wait complete. 2024-03-16T12:22:48.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.966+0000 7f3e1e600640 1 Processor -- start 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.966+0000 7f3e1e600640 1 -- start start 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.966+0000 7f3e1e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e1819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.966+0000 7f3e1e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e1e600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 0x7f3e181a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e1e600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18113a90 con 0x7f3e18077d80 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e1e600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18113910 con 0x7f3e180773e0 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e1e600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18113c10 con 0x7f3e180761a0 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54414/0 (socket says 172.21.15.98:54414) 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e16a00640 1 -- 172.21.15.98:0/1309305327 learned_addr learned my addr 172.21.15.98:0/1309305327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e1819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17e00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 0x7f3e181a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 -- 172.21.15.98:0/1309305327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 msgr2=0x7f3e1819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 -- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 msgr2=0x7f3e181a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 0x7f3e181a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e17400640 1 -- 172.21.15.98:0/1309305327 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e181a5c30 con 0x7f3e180761a0 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.967+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.968+0000 7f3e17e00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 0x7f3e181a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:48.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.968+0000 7f3e17400640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e1819e010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9f0 tx=0x7f3e0800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.968+0000 7f3e0fe00640 1 -- 172.21.15.98:0/1309305327 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800ccc0 con 0x7f3e180761a0 2024-03-16T12:22:48.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.968+0000 7f3e0fe00640 1 -- 172.21.15.98:0/1309305327 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08010910 con 0x7f3e180761a0 2024-03-16T12:22:48.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.968+0000 7f3e0fe00640 1 -- 172.21.15.98:0/1309305327 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08018960 con 0x7f3e180761a0 2024-03-16T12:22:48.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.969+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e181a5f20 con 0x7f3e180761a0 2024-03-16T12:22:48.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.969+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181a62d0 con 0x7f3e180761a0 2024-03-16T12:22:48.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.972+0000 7f3e0fe00640 1 -- 172.21.15.98:0/1309305327 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3e08010ac0 con 0x7f3e180761a0 2024-03-16T12:22:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.973+0000 7f3e0fe00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 0x7f3dfc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.973+0000 7f3e0fe00640 1 -- 172.21.15.98:0/1309305327 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e080cf1c0 con 0x7f3e180761a0 2024-03-16T12:22:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.973+0000 7f3e21436740 1 Processor -- start 2024-03-16T12:22:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.974+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 0x7f3dfc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.974+0000 7f3e21436740 1 -- start start 2024-03-16T12:22:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.974+0000 7f3e21436740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.974+0000 7f3e21436740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.974+0000 7f3e21436740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 0x55a2853fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54424/0 (socket says 172.21.15.98:54424) 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43856/0 (socket says 172.21.15.98:43856) 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e17400640 1 -- 172.21.15.98:0/4284596802 learned_addr learned my addr 172.21.15.98:0/4284596802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a2853be3b0 con 0x55a2853fc420 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a28535b450 con 0x55a2853f68c0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a285303cb0 con 0x55a2853f72b0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e17e00640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 0x55a2853fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.975+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 0x7f3dfc0a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3e1819f520 tx=0x7f3e04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 -- 172.21.15.98:0/4284596802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 msgr2=0x55a2853f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 -- 172.21.15.98:0/4284596802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 msgr2=0x55a2853fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 0x55a2853fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 -- 172.21.15.98:0/4284596802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2853251f0 con 0x55a2853f72b0 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e17400640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e17e00640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 0x55a2853fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e16a00640 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400f680 tx=0x7f3e0401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717444104a71dae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e0cc00640 1 -- 172.21.15.98:0/4284596802 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04005220 con 0x55a2853f72b0 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e0cc00640 1 -- 172.21.15.98:0/4284596802 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e04004c80 con 0x55a2853f72b0 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.976+0000 7f3e0cc00640 1 -- 172.21.15.98:0/4284596802 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0401cc20 con 0x55a2853f72b0 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 msgr2=0x55a2853fbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400f680 tx=0x7f3e0401c000 comp rx=0 tx=0).stop 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 shutdown_connections 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853fc420 0x55a2853fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a2853fbdb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 --2- 172.21.15.98:0/4284596802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.977+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 >> 172.21.15.98:0/4284596802 conn(0x55a2853f1e30 msgr2=0x55a2853f2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.978+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 shutdown_connections 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.978+0000 7f3e21436740 1 -- 172.21.15.98:0/4284596802 wait complete. 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.978+0000 7f3e21436740 1 -- start start 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.978+0000 7f3e21436740 1 Processor -- start 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 0x55a285400b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a2854080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a285317110 con 0x55a2853f68c0 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a2853574f0 con 0x55a2853fc420 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a2853ff3d0 con 0x55a2853f72b0 2024-03-16T12:22:48.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e21436740 10 client.0 _ll_register_callbacks cb 0x55a285304b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43864/0 (socket says 172.21.15.98:43864) 2024-03-16T12:22:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e16a00640 1 -- 172.21.15.98:0/3735248116 learned_addr learned my addr 172.21.15.98:0/3735248116 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e17400640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 0x55a285400b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.979+0000 7f3e17e00640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a2854080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 -- 172.21.15.98:0/3735248116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 msgr2=0x55a2854080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a2854080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 -- 172.21.15.98:0/3735248116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 msgr2=0x55a285400b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 0x55a285400b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 -- 172.21.15.98:0/3735248116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a28506f3f0 con 0x55a2853f72b0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e17400640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 0x55a285400b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e16a00640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e04011db0 tx=0x7f3e04005710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a16df195ec73858 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e01600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.980+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04033020 con 0x55a2853f72b0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e04004c80 con 0x55a2853f72b0 2024-03-16T12:22:48.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e0401cbb0 con 0x55a2853f72b0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e0403e050 con 0x55a2853f72b0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e21436740 10 client.25717 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a2851397d0 con 0x55a2853f72b0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e04004c80 con 0x55a2853f72b0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.981+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3e04023070 con 0x55a2853f72b0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 10 client.25717 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 10 client.25717 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 20 client.25717 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a28540caa0 0x55a28540eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 4 client.25717 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a285368990 con 0x55a28540caa0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e17400640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a28540caa0 0x55a28540eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.982+0000 7f3e01600640 10 client.25717 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.983+0000 7f3e17400640 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a28540caa0 0x55a28540eee0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c817fa989c2dd751 server_cookie=9d75f2a45f44be1c in_seq=0 out_seq=0 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.984+0000 7f3e01600640 1 -- 172.21.15.98:0/3735248116 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a285368990 con 0x55a28540caa0 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.984+0000 7f3e01600640 10 client.25717 handle_command_reply: tid=0 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.984+0000 7f3e21436740 1 client.25717 shutdown 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.984+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 msgr2=0x55a285401080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.984+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e04011db0 tx=0x7f3e04005710 comp rx=0 tx=0).stop 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 shutdown_connections 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a28540caa0 0x55a28540eee0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a2854080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f72b0 0x55a285401080 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 --2- 172.21.15.98:0/3735248116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f68c0 0x55a285400b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 >> 172.21.15.98:0/3735248116 conn(0x55a2853f1e30 msgr2=0x55a2853f11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e01600640 0 client.25717 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 shutdown_connections 2024-03-16T12:22:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.985+0000 7f3e21436740 1 -- 172.21.15.98:0/3735248116 wait complete. 2024-03-16T12:22:48.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:48.986+0000 7f3e21436740 20 client.25717 trim_cache size 0 max 16384 2024-03-16T12:22:48.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.004+0000 7f3e21436740 1 Processor -- start 2024-03-16T12:22:48.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.004+0000 7f3e21436740 1 -- start start 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a285408080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853fc420 0x55a2853f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b31d0 con 0x55a2853f72b0 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b0aa0 con 0x55a2853f68c0 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e21436740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181ae8f0 con 0x55a2853fc420 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54442/0 (socket says 172.21.15.98:54442) 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e17400640 1 -- 172.21.15.98:0/217555096 learned_addr learned my addr 172.21.15.98:0/217555096 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e17e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853fc420 0x55a2853f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.005+0000 7f3e16a00640 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a285408080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 -- 172.21.15.98:0/217555096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853fc420 msgr2=0x55a2853f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853fc420 0x55a2853f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 -- 172.21.15.98:0/217555096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 msgr2=0x55a285408080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a285408080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 -- 172.21.15.98:0/217555096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2853edc10 con 0x55a2853f68c0 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e16a00640 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a285408080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e17400640 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3e0809b170 tx=0x7f3e080cc7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f53d8ab361d2855 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e03e00640 1 -- 172.21.15.98:0/217555096 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080b4070 con 0x55a2853f68c0 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e03e00640 1 -- 172.21.15.98:0/217555096 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e080a3f40 con 0x55a2853f68c0 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.006+0000 7f3e03e00640 1 -- 172.21.15.98:0/217555096 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080afb80 con 0x55a2853f68c0 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 -- 172.21.15.98:0/217555096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 msgr2=0x55a2853f0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3e0809b170 tx=0x7f3e080cc7d0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 -- 172.21.15.98:0/217555096 shutdown_connections 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853fc420 0x55a2853f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a285408080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 --2- 172.21.15.98:0/217555096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853f68c0 0x55a2853f0960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.007+0000 7f3e21436740 1 -- 172.21.15.98:0/217555096 >> 172.21.15.98:0/217555096 conn(0x55a2853f1e30 msgr2=0x55a2853f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.008+0000 7f3e21436740 1 -- 172.21.15.98:0/217555096 shutdown_connections 2024-03-16T12:22:48.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.008+0000 7f3e21436740 1 -- 172.21.15.98:0/217555096 wait complete. 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.008+0000 7f3e21436740 1 -- start start 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.008+0000 7f3e21436740 1 Processor -- start 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a2854054e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a285470e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181aef60 con 0x55a2853f72b0 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b03d0 con 0x55a2853fc420 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181af630 con 0x55a2853f68c0 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e21436740 10 client.0 _ll_register_callbacks cb 0x7f3e04004a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e17400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a2854054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e17400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43892/0 (socket says 172.21.15.98:43892) 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e17400640 1 -- 172.21.15.98:0/1257098696 learned_addr learned my addr 172.21.15.98:0/1257098696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.009+0000 7f3e17e00640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a285470e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 -- 172.21.15.98:0/1257098696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 msgr2=0x55a285470e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a285470e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 -- 172.21.15.98:0/1257098696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 msgr2=0x55a2854054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a2854054e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 -- 172.21.15.98:0/1257098696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a285471350 con 0x55a2853f68c0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17e00640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a285470e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a2854054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e17400640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9c0 tx=0x7f3e0800c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fece754f4dbbff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e01600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.010+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080b4040 con 0x55a2853f68c0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e08099aa0 con 0x55a2853f68c0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e080a3f40 con 0x55a2853f68c0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e080afb80 con 0x55a2853f68c0 2024-03-16T12:22:48.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 10 client.25723 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a2854725f0 con 0x55a2853f68c0 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e080a3f40 con 0x55a2853f68c0 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3e08018b00 con 0x55a2853f68c0 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 10 client.25723 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 10 client.25723 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 20 client.25723 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.011+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a285472f40 0x55a285475360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e21436740 4 client.25723 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a285362ce0 con 0x55a285472f40 2024-03-16T12:22:48.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a285472f40 0x55a285475360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e16a00640 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a285472f40 0x55a285475360 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6d078dde6204911 server_cookie=a515f9a3f71bcbef in_seq=0 out_seq=0 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e01600640 10 client.25723 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e01600640 1 -- 172.21.15.98:0/1257098696 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a285362ce0 con 0x55a285472f40 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.012+0000 7f3e01600640 10 client.25723 handle_command_reply: tid=0 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 client.25723 shutdown 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 msgr2=0x55a285404fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9c0 tx=0x7f3e0800c7d0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 shutdown_connections 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a285472f40 0x55a285475360 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a2853fc420 0x55a285470e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a2853f72b0 0x55a2854054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 --2- 172.21.15.98:0/1257098696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a2853f68c0 0x55a285404fa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 >> 172.21.15.98:0/1257098696 conn(0x55a2853f1e30 msgr2=0x55a2853f5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e01600640 0 client.25723 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 shutdown_connections 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 1 -- 172.21.15.98:0/1257098696 wait complete. 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.013+0000 7f3e21436740 20 client.25723 trim_cache size 0 max 16384 2024-03-16T12:22:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 msgr2=0x7f3dfc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 0x7f3dfc0a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3e1819f520 tx=0x7f3e04005f90 comp rx=0 tx=0).stop 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 msgr2=0x7f3e1819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e1819e010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9f0 tx=0x7f3e0800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 shutdown_connections 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3dfc0a2880 0x7f3dfc0a4d40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e18077d80 0x7f3e181a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e180773e0 0x7f3e1819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 --2- 172.21.15.98:0/1309305327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e180761a0 0x7f3e1819e010 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 >> 172.21.15.98:0/1309305327 conn(0x7f3e180fef90 msgr2=0x7f3e18100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 shutdown_connections 2024-03-16T12:22:49.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.015+0000 7f3e1e600640 1 -- 172.21.15.98:0/1309305327 wait complete. 2024-03-16T12:22:49.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:48 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2928585052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:49.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 -- 172.21.15.98:0/4118601922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50300773e0 msgr2=0x7f5030077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 --2- 172.21.15.98:0/4118601922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50300773e0 0x7f5030077840 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f501c009900 tx=0x7f501c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 -- 172.21.15.98:0/4118601922 shutdown_connections 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 --2- 172.21.15.98:0/4118601922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5030077d80 0x7f5030078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 --2- 172.21.15.98:0/4118601922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50300773e0 0x7f5030077840 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 --2- 172.21.15.98:0/4118601922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f50300765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.217+0000 7f5036800640 1 -- 172.21.15.98:0/4118601922 >> 172.21.15.98:0/4118601922 conn(0x7f50300fef90 msgr2=0x7f50301013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.218+0000 7f5036800640 1 -- 172.21.15.98:0/4118601922 shutdown_connections 2024-03-16T12:22:49.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.218+0000 7f5036800640 1 -- 172.21.15.98:0/4118601922 wait complete. 2024-03-16T12:22:49.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 Processor -- start 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 -- start start 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50300773e0 0x7f503019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5030113ad0 con 0x7f50300773e0 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5030113950 con 0x7f50300761a0 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.219+0000 7f5036800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5030113c50 con 0x7f5030077d80 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54454/0 (socket says 172.21.15.98:54454) 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43906/0 (socket says 172.21.15.98:43906) 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502f400640 1 -- 172.21.15.98:0/2574036702 learned_addr learned my addr 172.21.15.98:0/2574036702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502ea00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50300773e0 0x7f503019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 -- 172.21.15.98:0/2574036702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 msgr2=0x7f503019e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 -- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50300773e0 msgr2=0x7f503019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50300773e0 0x7f503019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 -- 172.21.15.98:0/2574036702 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5030103560 con 0x7f5030077d80 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502f400640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.220+0000 7f502fe00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f502800da10 tx=0x7f502800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.221+0000 7f5027e00640 1 -- 172.21.15.98:0/2574036702 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502800bd70 con 0x7f5030077d80 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.221+0000 7f5027e00640 1 -- 172.21.15.98:0/2574036702 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5028003e50 con 0x7f5030077d80 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.221+0000 7f5027e00640 1 -- 172.21.15.98:0/2574036702 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50280106a0 con 0x7f5030077d80 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.221+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5030103850 con 0x7f5030077d80 2024-03-16T12:22:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.221+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5030103d20 con 0x7f5030077d80 2024-03-16T12:22:49.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.224+0000 7f5027e00640 1 -- 172.21.15.98:0/2574036702 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5028003970 con 0x7f5030077d80 2024-03-16T12:22:49.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.225+0000 7f50395e8740 1 Processor -- start 2024-03-16T12:22:49.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f5027e00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 -- start start 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f5027e00640 1 -- 172.21.15.98:0/2574036702 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f50280cf240 con 0x7f5030077d80 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680dece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559e680a63b0 con 0x559e680de8c0 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559e68043450 con 0x559e680e4420 2024-03-16T12:22:49.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f50395e8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559e67febcb0 con 0x559e680df2b0 2024-03-16T12:22:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.226+0000 7f502ea00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54466/0 (socket says 172.21.15.98:54466) 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 -- 172.21.15.98:0/4176789512 learned_addr learned my addr 172.21.15.98:0/4176789512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502f400640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502ea00640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 -- 172.21.15.98:0/4176789512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 msgr2=0x559e680e3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 -- 172.21.15.98:0/4176789512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 msgr2=0x559e680dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680dece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502fe00640 1 -- 172.21.15.98:0/4176789512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6800d1f0 con 0x559e680e4420 2024-03-16T12:22:49.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.227+0000 7f502f400640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.228+0000 7f502fe00640 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f502809b520 tx=0x7f502809b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4cfbfdf3352649 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.228+0000 7f502ea00640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 0x7f50140a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f501c02c240 tx=0x7f501c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.228+0000 7f5024c00640 1 -- 172.21.15.98:0/4176789512 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502809f970 con 0x559e680e4420 2024-03-16T12:22:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.228+0000 7f5024c00640 1 -- 172.21.15.98:0/4176789512 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50280a3fc0 con 0x559e680e4420 2024-03-16T12:22:49.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.228+0000 7f5024c00640 1 -- 172.21.15.98:0/4176789512 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028097e90 con 0x559e680e4420 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 -- 172.21.15.98:0/4176789512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 msgr2=0x559e680e68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f502809b520 tx=0x7f502809b720 comp rx=0 tx=0).stop 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 -- 172.21.15.98:0/4176789512 shutdown_connections 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680e68b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 --2- 172.21.15.98:0/4176789512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 -- 172.21.15.98:0/4176789512 >> 172.21.15.98:0/4176789512 conn(0x559e680d9e30 msgr2=0x559e680da230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.229+0000 7f50395e8740 1 -- 172.21.15.98:0/4176789512 shutdown_connections 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.230+0000 7f50395e8740 1 -- 172.21.15.98:0/4176789512 wait complete. 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.230+0000 7f50395e8740 1 -- start start 2024-03-16T12:22:49.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.230+0000 7f50395e8740 1 Processor -- start 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680f0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559e67fff110 con 0x559e680de8c0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559e6803f4f0 con 0x559e680e4420 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559e680e73d0 con 0x559e680df2b0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.231+0000 7f50395e8740 10 client.0 _ll_register_callbacks cb 0x559e67fecb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43922/0 (socket says 172.21.15.98:43922) 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 -- 172.21.15.98:0/2003976583 learned_addr learned my addr 172.21.15.98:0/2003976583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502f400640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502fe00640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 -- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 msgr2=0x559e680f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 -- 172.21.15.98:0/2003976583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 msgr2=0x559e680e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680e8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 -- 172.21.15.98:0/2003976583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e67d573f0 con 0x559e680df2b0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f502ea00640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f501c000c00 tx=0x7f501c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d8a1b7f55ffe9f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50280a3fc0 con 0x559e680df2b0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.232+0000 7f5019600640 10 client.25735 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.233+0000 7f5019600640 1 -- 172.21.15.98:0/2003976583 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c052020 con 0x559e680df2b0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.233+0000 7f5019600640 1 -- 172.21.15.98:0/2003976583 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f501c0046c0 con 0x559e680df2b0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.233+0000 7f5019600640 1 -- 172.21.15.98:0/2003976583 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f501c004920 con 0x559e680df2b0 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.233+0000 7f50395e8740 10 client.25735 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.233+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=24}) -- 0x559e67e217d0 con 0x559e680df2b0 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f5019600640 1 -- 172.21.15.98:0/2003976583 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f501c040040 con 0x559e680df2b0 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 10 client.25735 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 20 client.25735 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e680f4970 0x559e680f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 4 client.25735 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e68050990 con 0x559e680f4970 2024-03-16T12:22:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.234+0000 7f502f400640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e680f4970 0x559e680f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.235+0000 7f502f400640 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e680f4970 0x559e680f6da0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec427da05191858 server_cookie=54d3c0c7e0dd6f6c in_seq=0 out_seq=0 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.235+0000 7f5019600640 10 client.25735 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.235+0000 7f5019600640 1 -- 172.21.15.98:0/2003976583 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e68050990 con 0x559e680f4970 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.235+0000 7f5019600640 10 client.25735 handle_command_reply: tid=0 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.235+0000 7f50395e8740 1 client.25735 shutdown 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 msgr2=0x559e680e9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f501c000c00 tx=0x7f501c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 shutdown_connections 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680e4420 0x559e680f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e680f4970 0x559e680f6da0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f5019600640 0 client.25735 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680e9130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 --2- 172.21.15.98:0/2003976583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680de8c0 0x559e680e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 >> 172.21.15.98:0/2003976583 conn(0x559e680d9e30 msgr2=0x559e680d91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 shutdown_connections 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 1 -- 172.21.15.98:0/2003976583 wait complete. 2024-03-16T12:22:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.236+0000 7f50395e8740 20 client.25735 trim_cache size 0 max 16384 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.250+0000 7f50395e8740 1 Processor -- start 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 -- start start 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680de8c0 0x559e680e0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680f0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f502ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f502ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54488/0 (socket says 172.21.15.98:54488) 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50301bb8f0 con 0x559e680e4420 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50301b91c0 con 0x559e680df2b0 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.251+0000 7f50395e8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50301b7010 con 0x559e680de8c0 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 -- 172.21.15.98:0/2438220325 learned_addr learned my addr 172.21.15.98:0/2438220325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502f400640 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680de8c0 0x559e680e0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502fe00640 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 -- 172.21.15.98:0/2438220325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680de8c0 msgr2=0x559e680e0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680de8c0 0x559e680e0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 -- 172.21.15.98:0/2438220325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 msgr2=0x559e680f0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680f0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.252+0000 7f502ea00640 1 -- 172.21.15.98:0/2438220325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e680d5c10 con 0x559e680df2b0 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.253+0000 7f502ea00640 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f501c009230 tx=0x7f501c04b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45a4543338c0424 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.253+0000 7f501be00640 1 -- 172.21.15.98:0/2438220325 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c052070 con 0x559e680df2b0 2024-03-16T12:22:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.253+0000 7f501be00640 1 -- 172.21.15.98:0/2438220325 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50280a3fc0 con 0x559e680df2b0 2024-03-16T12:22:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.253+0000 7f501be00640 1 -- 172.21.15.98:0/2438220325 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c003d30 con 0x559e680df2b0 2024-03-16T12:22:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.254+0000 7f50395e8740 1 -- 172.21.15.98:0/2438220325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 msgr2=0x559e680e0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.254+0000 7f50395e8740 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f501c009230 tx=0x7f501c04b000 comp rx=0 tx=0).stop 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 -- 172.21.15.98:0/2438220325 shutdown_connections 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680f0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680df2b0 0x559e680e0c90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 --2- 172.21.15.98:0/2438220325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680de8c0 0x559e680e0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 -- 172.21.15.98:0/2438220325 >> 172.21.15.98:0/2438220325 conn(0x559e680d9e30 msgr2=0x559e680da210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 -- 172.21.15.98:0/2438220325 shutdown_connections 2024-03-16T12:22:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 -- 172.21.15.98:0/2438220325 wait complete. 2024-03-16T12:22:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.255+0000 7f50395e8740 1 -- start start 2024-03-16T12:22:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 Processor -- start 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680ee110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680ef120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50301b7680 con 0x559e680e4420 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50301b8af0 con 0x559e680de8c0 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f50301b7d50 con 0x559e680df2b0 2024-03-16T12:22:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.256+0000 7f50395e8740 10 client.0 _ll_register_callbacks cb 0x7f501c047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54496/0 (socket says 172.21.15.98:54496) 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 -- 172.21.15.98:0/3694684834 learned_addr learned my addr 172.21.15.98:0/3694684834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502ea00640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502fe00640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 -- 172.21.15.98:0/3694684834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 msgr2=0x559e680ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 -- 172.21.15.98:0/3694684834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 msgr2=0x559e680ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.257+0000 7f502f400640 1 -- 172.21.15.98:0/3694684834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e680ef710 con 0x559e680de8c0 2024-03-16T12:22:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.258+0000 7f502f400640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5020011670 tx=0x7f5020011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ff8cfa6302fb315 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.258+0000 7f5019600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.258+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020010d10 con 0x559e680de8c0 2024-03-16T12:22:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.258+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5020005020 con 0x559e680de8c0 2024-03-16T12:22:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.258+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50280a3fc0 con 0x559e680de8c0 2024-03-16T12:22:49.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.260+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5020030050 con 0x559e680de8c0 2024-03-16T12:22:49.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.260+0000 7f50395e8740 10 client.25751 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:49.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.260+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559e6815b000 con 0x559e680de8c0 2024-03-16T12:22:49.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.260+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50280a3fc0 con 0x559e680de8c0 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5020028430 con 0x559e680de8c0 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 10 client.25751 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 20 client.25751 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e6815ba30 0x559e6815de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 4 client.25751 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e6804ace0 con 0x559e6815ba30 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.261+0000 7f502ea00640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e6815ba30 0x559e6815de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f502ea00640 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e6815ba30 0x559e6815de50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15f852c46c4116 server_cookie=fd84ad142cef5b9c in_seq=0 out_seq=0 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f5019600640 10 client.25751 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f5019600640 1 -- 172.21.15.98:0/3694684834 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e6804ace0 con 0x559e6815ba30 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f5019600640 10 client.25751 handle_command_reply: tid=0 2024-03-16T12:22:49.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f50395e8740 1 client.25751 shutdown 2024-03-16T12:22:49.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 msgr2=0x559e680edbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5020011670 tx=0x7f5020011b50 comp rx=0 tx=0).stop 2024-03-16T12:22:49.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.262+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 shutdown_connections 2024-03-16T12:22:49.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559e680e4420 0x559e680ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x559e6815ba30 0x559e6815de50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559e680df2b0 0x559e680ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 --2- 172.21.15.98:0/3694684834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559e680de8c0 0x559e680edbd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 >> 172.21.15.98:0/3694684834 conn(0x559e680d9e30 msgr2=0x559e680e8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f5019600640 0 client.25751 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 shutdown_connections 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 1 -- 172.21.15.98:0/3694684834 wait complete. 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.263+0000 7f50395e8740 20 client.25751 trim_cache size 0 max 16384 2024-03-16T12:22:49.246 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 msgr2=0x7f50140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 0x7f50140a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f501c02c240 tx=0x7f501c005fd0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 msgr2=0x7f50301a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f502800da10 tx=0x7f502800ded0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 shutdown_connections 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5030077d80 0x7f50301a56a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50300773e0 0x7f503019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.264+0000 7f5036800640 1 --2- 172.21.15.98:0/2574036702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50300761a0 0x7f503019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:49.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.265+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 >> 172.21.15.98:0/2574036702 conn(0x7f50300fef90 msgr2=0x7f5030100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:49.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.265+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 shutdown_connections 2024-03-16T12:22:49.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:49.265+0000 7f5036800640 1 -- 172.21.15.98:0/2574036702 wait complete. 2024-03-16T12:22:49.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:48 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2928585052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:49.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:48 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2928585052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:22:50.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:49 smithi098 ceph-mon[31633]: pgmap v444: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:22:50.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:49 smithi153 ceph-mon[36901]: pgmap v444: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:22:50.273 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T12:22:50.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:22:50.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T12:22:50.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:49 smithi173 ceph-mon[36632]: pgmap v444: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 -- 172.21.15.98:0/1242904057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 msgr2=0x7fdcb8104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1242904057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8104530 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009860 tx=0x7fdca802bb70 comp rx=0 tx=0).stop 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 -- 172.21.15.98:0/1242904057 shutdown_connections 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1242904057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 0x7fdcb81097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1242904057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 0x7fdcb8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1242904057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8104530 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 -- 172.21.15.98:0/1242904057 >> 172.21.15.98:0/1242904057 conn(0x7fdcb80fbb50 msgr2=0x7fdcb80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 -- 172.21.15.98:0/1242904057 shutdown_connections 2024-03-16T12:22:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.472+0000 7fdcb7200640 1 -- 172.21.15.98:0/1242904057 wait complete. 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.473+0000 7fdcb7200640 1 Processor -- start 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.473+0000 7fdcb7200640 1 -- start start 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 0x7fdcb81000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 0x7fdcb8100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810d350 con 0x7fdcb81073a0 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810d1d0 con 0x7fdcb8104a70 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810d4d0 con 0x7fdcb8102140 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43944/0 (socket says 172.21.15.98:43944) 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 -- 172.21.15.98:0/1330718052 learned_addr learned my addr 172.21.15.98:0/1330718052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5400640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 0x7fdcb81000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb6800640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 0x7fdcb8100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 -- 172.21.15.98:0/1330718052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 msgr2=0x7fdcb81000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 0x7fdcb81000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 -- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 msgr2=0x7fdcb8100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 0x7fdcb8100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.474+0000 7fdcb5e00640 1 -- 172.21.15.98:0/1330718052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb8100ec0 con 0x7fdcb8102140 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdca80349e0 tx=0x7fdca8032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcaea00640 1 -- 172.21.15.98:0/1330718052 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8041030 con 0x7fdcb8102140 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcaea00640 1 -- 172.21.15.98:0/1330718052 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdca802c510 con 0x7fdcb8102140 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcaea00640 1 -- 172.21.15.98:0/1330718052 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8039040 con 0x7fdcb8102140 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb81a1ac0 con 0x7fdcb8102140 2024-03-16T12:22:50.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.475+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb81a1fc0 con 0x7fdcb8102140 2024-03-16T12:22:50.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 Processor -- start 2024-03-16T12:22:50.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcaea00640 1 -- 172.21.15.98:0/1330718052 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdca8044070 con 0x7fdcb8102140 2024-03-16T12:22:50.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 -- start start 2024-03-16T12:22:50.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e6370 0x55c71b3eae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3ed970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c71b348f80 con 0x55c71b3eb4e0 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.480+0000 7fdcbf792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c71b2f1ad0 con 0x55c71b3e6370 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcbf792740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c71b345020 con 0x55c71b3e5980 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43948/0 (socket says 172.21.15.98:43948) 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 -- 172.21.15.98:0/201675441 learned_addr learned my addr 172.21.15.98:0/201675441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e6370 0x55c71b3eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb5400640 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 -- 172.21.15.98:0/201675441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e6370 msgr2=0x55c71b3eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e6370 0x55c71b3eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 -- 172.21.15.98:0/201675441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 msgr2=0x55c71b3ed970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3ed970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.481+0000 7fdcb6800640 1 -- 172.21.15.98:0/201675441 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c71b3096f0 con 0x55c71b3e5980 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdcb6800640 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009fd0 tx=0x7fdcb0010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e42b4fb4230b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdcaea00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 0x7fdc9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdcaea00640 1 -- 172.21.15.98:0/1330718052 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdca80f02d0 con 0x7fdcb8102140 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 0x7fdc9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdca3000640 1 -- 172.21.15.98:0/201675441 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0011700 con 0x55c71b3e5980 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdca3000640 1 -- 172.21.15.98:0/201675441 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb0011d80 con 0x55c71b3e5980 2024-03-16T12:22:50.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.482+0000 7fdca3000640 1 -- 172.21.15.98:0/201675441 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb001a380 con 0x55c71b3e5980 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.483+0000 7fdcb5e00640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 0x7fdc9c0a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdca80edde0 tx=0x7fdca80f3db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.483+0000 7fdcbf792740 1 -- 172.21.15.98:0/201675441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 msgr2=0x55c71b3e5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.483+0000 7fdcbf792740 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009fd0 tx=0x7fdcb0010860 comp rx=0 tx=0).stop 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 -- 172.21.15.98:0/201675441 shutdown_connections 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3ed970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e6370 0x55c71b3eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 --2- 172.21.15.98:0/201675441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e5da0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 -- 172.21.15.98:0/201675441 >> 172.21.15.98:0/201675441 conn(0x55c71b3ad930 msgr2=0x55c71b3e12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 -- 172.21.15.98:0/201675441 shutdown_connections 2024-03-16T12:22:50.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 -- 172.21.15.98:0/201675441 wait complete. 2024-03-16T12:22:50.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.484+0000 7fdcbf792740 1 -- start start 2024-03-16T12:22:50.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 Processor -- start 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3efd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 0x55c71b3f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c71b3ee490 con 0x55c71b3eb4e0 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c71b304f30 con 0x55c71b3e5980 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c71b3ee610 con 0x55c71b3e6370 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcbf792740 10 client.0 _ll_register_callbacks cb 0x55c71b2f2bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.485+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3efd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54534/0 (socket says 172.21.15.98:54534) 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 0x55c71b3f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb6800640 1 -- 172.21.15.98:0/3908474138 learned_addr learned my addr 172.21.15.98:0/3908474138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 -- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 msgr2=0x55c71b3efd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3efd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 -- 172.21.15.98:0/3908474138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 msgr2=0x55c71b3f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 -- 172.21.15.98:0/3908474138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c71b1257e0 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 0x55c71b3f0260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8101820 tx=0x7fdca4011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada22567aa0585b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdc97400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.486+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca401e020 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdca400faa0 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcb0011d80 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdca400fca0 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdcbf792740 10 client.25753 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c71b185f70 con 0x55c71b3e6370 2024-03-16T12:22:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.487+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb0011d80 con 0x55c71b3e6370 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdca4019070 con 0x55c71b3e6370 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 10 client.25753 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 10 client.25753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 20 client.25753 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b3fbc30 0x55c71b3fe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 4 client.25753 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:50.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c71b3564c0 con 0x55c71b3fbc30 2024-03-16T12:22:50.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.488+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b3fbc30 0x55c71b3fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.489+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b3fbc30 0x55c71b3fe070 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f8cdb88330fb791 server_cookie=c6e97edffb6ea946 in_seq=0 out_seq=0 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.490+0000 7fdc97400640 10 client.25753 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.490+0000 7fdc97400640 1 -- 172.21.15.98:0/3908474138 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c71b3564c0 con 0x55c71b3fbc30 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.490+0000 7fdc97400640 10 client.25753 handle_command_reply: tid=0 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.490+0000 7fdcbf792740 1 client.25753 shutdown 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 msgr2=0x55c71b3f0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 0x55c71b3f0260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8101820 tx=0x7fdca4011b50 comp rx=0 tx=0).stop 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 shutdown_connections 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b3fbc30 0x55c71b3fe070 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e6370 0x55c71b3f0260 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3908474138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 >> 172.21.15.98:0/3908474138 conn(0x55c71b3ad930 msgr2=0x55c71b3e0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdc97400640 0 client.25753 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 shutdown_connections 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 1 -- 172.21.15.98:0/3908474138 wait complete. 2024-03-16T12:22:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.491+0000 7fdcbf792740 20 client.25753 trim_cache size 0 max 16384 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.525+0000 7fdcbf792740 1 Processor -- start 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 -- start start 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3eb4e0 0x55c71b402930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81aec80 con 0x55c71b3e6370 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81ac550 con 0x55c71b3eb4e0 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcbf792740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81aa3a0 con 0x55c71b3e5980 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb5400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43972/0 (socket says 172.21.15.98:43972) 2024-03-16T12:22:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb6800640 1 -- 172.21.15.98:0/525617264 learned_addr learned my addr 172.21.15.98:0/525617264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb5400640 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47400/0 (socket says 172.21.15.98:47400) 2024-03-16T12:22:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.526+0000 7fdcb6800640 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3eb4e0 0x55c71b402930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 -- 172.21.15.98:0/525617264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3eb4e0 msgr2=0x55c71b402930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3eb4e0 0x55c71b402930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 -- 172.21.15.98:0/525617264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 msgr2=0x55c71b3f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 -- 172.21.15.98:0/525617264 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c71b01d170 con 0x55c71b3e5980 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdcb6800640 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00099a0 tx=0x7fdcb0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63086062e9bff1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdca2600640 1 -- 172.21.15.98:0/525617264 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb001caa0 con 0x55c71b3e5980 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdca2600640 1 -- 172.21.15.98:0/525617264 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb002e520 con 0x55c71b3e5980 2024-03-16T12:22:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.527+0000 7fdca2600640 1 -- 172.21.15.98:0/525617264 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0011ac0 con 0x55c71b3e5980 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 -- 172.21.15.98:0/525617264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 msgr2=0x55c71b3e03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00099a0 tx=0x7fdcb0005fd0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 -- 172.21.15.98:0/525617264 shutdown_connections 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3eb4e0 0x55c71b402930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 --2- 172.21.15.98:0/525617264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3e5980 0x55c71b3e03f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.528+0000 7fdcbf792740 1 -- 172.21.15.98:0/525617264 >> 172.21.15.98:0/525617264 conn(0x55c71b3ad930 msgr2=0x55c71b3dfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.529+0000 7fdcbf792740 1 -- 172.21.15.98:0/525617264 shutdown_connections 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.529+0000 7fdcbf792740 1 -- 172.21.15.98:0/525617264 wait complete. 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.529+0000 7fdcbf792740 1 -- start start 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.529+0000 7fdcbf792740 1 Processor -- start 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81aaa10 con 0x55c71b3e6370 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81abe80 con 0x55c71b3e5980 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb81ab0e0 con 0x55c71b3eb4e0 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcbf792740 10 client.0 _ll_register_callbacks cb 0x7fdca4009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcb6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54544/0 (socket says 172.21.15.98:54544) 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcb6800640 1 -- 172.21.15.98:0/3301793334 learned_addr learned my addr 172.21.15.98:0/3301793334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.530+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 -- 172.21.15.98:0/3301793334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3eb4e0 msgr2=0x55c71b3f5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 -- 172.21.15.98:0/3301793334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 msgr2=0x55c71b3f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 -- 172.21.15.98:0/3301793334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c71b3f58d0 con 0x55c71b3e5980 2024-03-16T12:22:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcb6800640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009de0 tx=0x7fdcb0009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2857c37e9268eabf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdc97400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:22:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb001caa0 con 0x55c71b3e5980 2024-03-16T12:22:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcb002e520 con 0x55c71b3e5980 2024-03-16T12:22:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdcb002e040 con 0x55c71b3e5980 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdcb002e240 con 0x55c71b3e5980 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcbf792740 10 client.25769 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.531+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c71b462a30 con 0x55c71b3e5980 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb002e520 con 0x55c71b3e5980 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdcb0039040 con 0x55c71b3e5980 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 10 client.25769 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 20 client.25769 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b4631a0 0x55c71b4655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 4 client.25769 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c71b3efb10 con 0x55c71b4631a0 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.532+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b4631a0 0x55c71b4655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.533+0000 7fdc97400640 10 client.25769 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.533+0000 7fdcb5400640 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b4631a0 0x55c71b4655c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3148b86e75980f8 server_cookie=a92df9b2ee138178 in_seq=0 out_seq=0 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdc97400640 1 -- 172.21.15.98:0/3301793334 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c71b3efb10 con 0x55c71b4631a0 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdc97400640 10 client.25769 handle_command_reply: tid=0 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 client.25769 shutdown 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 msgr2=0x55c71b3f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009de0 tx=0x7fdcb0009590 comp rx=0 tx=0).stop 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 shutdown_connections 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c71b4631a0 0x55c71b4655c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c71b3eb4e0 0x55c71b3f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c71b3e6370 0x55c71b3f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 --2- 172.21.15.98:0/3301793334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c71b3e5980 0x55c71b3f6b30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 >> 172.21.15.98:0/3301793334 conn(0x55c71b3ad930 msgr2=0x55c71b3e0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdc97400640 0 client.25769 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 shutdown_connections 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 1 -- 172.21.15.98:0/3301793334 wait complete. 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.536+0000 7fdcbf792740 20 client.25769 trim_cache size 0 max 16384 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-03-16T12:22:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "00698bdf-8251-490e-9205-9a45c75a4b83", 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 msgr2=0x7fdc9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 0x7fdc9c0a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdca80edde0 tx=0x7fdca80f3db0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 msgr2=0x7fdcb8101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdca80349e0 tx=0x7fdca8032450 comp rx=0 tx=0).stop 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 shutdown_connections 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdc9c0a2880 0x7fdc9c0a4d40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcb81073a0 0x7fdcb8100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcb8104a70 0x7fdcb81000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 --2- 172.21.15.98:0/1330718052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8102140 0x7fdcb8101ab0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.537+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 >> 172.21.15.98:0/1330718052 conn(0x7fdcb80fbb50 msgr2=0x7fdcb81029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.538+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 shutdown_connections 2024-03-16T12:22:50.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.538+0000 7fdcb7200640 1 -- 172.21.15.98:0/1330718052 wait complete. 2024-03-16T12:22:50.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:22:50.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.745+0000 7f3ca2200640 1 -- 172.21.15.98:0/2467795492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 msgr2=0x7f3c9c104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.745+0000 7f3ca2200640 1 --2- 172.21.15.98:0/2467795492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c104d10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009900 tx=0x7f3c8802bc30 comp rx=0 tx=0).stop 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 -- 172.21.15.98:0/2467795492 shutdown_connections 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 --2- 172.21.15.98:0/2467795492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c105250 0x7f3c9c1056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 --2- 172.21.15.98:0/2467795492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c104d10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 --2- 172.21.15.98:0/2467795492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 -- 172.21.15.98:0/2467795492 >> 172.21.15.98:0/2467795492 conn(0x7f3c9c0fbb50 msgr2=0x7f3c9c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 -- 172.21.15.98:0/2467795492 shutdown_connections 2024-03-16T12:22:50.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.746+0000 7f3ca2200640 1 -- 172.21.15.98:0/2467795492 wait complete. 2024-03-16T12:22:50.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 Processor -- start 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 -- start start 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c105250 0x7f3c9c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c113910 con 0x7f3c9c0fff30 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c113790 con 0x7f3c9c1028a0 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.747+0000 7f3ca2200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c113a90 con 0x7f3c9c105250 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c19dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47428/0 (socket says 172.21.15.98:47428) 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3ca0e00640 1 -- 172.21.15.98:0/1191581316 learned_addr learned my addr 172.21.15.98:0/1191581316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3ca1800640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c105250 0x7f3c9c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 -- 172.21.15.98:0/1191581316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c105250 msgr2=0x7f3c9c1a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c105250 0x7f3c9c1a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 -- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 msgr2=0x7f3c9c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 -- 172.21.15.98:0/1191581316 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c1a5bd0 con 0x7f3c9c1028a0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.748+0000 7f3c9be00640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c19e4f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009230 tx=0x7f3c8802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.749+0000 7f3c99600640 1 -- 172.21.15.98:0/1191581316 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88041070 con 0x7f3c9c1028a0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.749+0000 7f3c99600640 1 -- 172.21.15.98:0/1191581316 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c88004450 con 0x7f3c9c1028a0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.749+0000 7f3c99600640 1 -- 172.21.15.98:0/1191581316 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88034440 con 0x7f3c9c1028a0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.749+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9c1a5e60 con 0x7f3c9c1028a0 2024-03-16T12:22:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.749+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9c1a6270 con 0x7f3c9c1028a0 2024-03-16T12:22:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.753+0000 7f3ca6627740 1 Processor -- start 2024-03-16T12:22:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.753+0000 7f3ca6627740 1 -- start start 2024-03-16T12:22:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3c99600640 1 -- 172.21.15.98:0/1191581316 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3c88044070 con 0x7f3c9c1028a0 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca52ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca531f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca534a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5617ca4911f0 con 0x5617ca532570 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617ca439a80 con 0x5617ca52ca10 2024-03-16T12:22:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca6627740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5617ca48d290 con 0x5617ca52d400 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca1800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca52ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca1800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca52ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54566/0 (socket says 172.21.15.98:54566) 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca1800640 1 -- 172.21.15.98:0/4266246381 learned_addr learned my addr 172.21.15.98:0/4266246381 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3c9be00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca534a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.754+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca531f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 -- 172.21.15.98:0/4266246381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 msgr2=0x5617ca52ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca52ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 -- 172.21.15.98:0/4266246381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 msgr2=0x5617ca534a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca534a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 -- 172.21.15.98:0/4266246381 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617ca26f7d0 con 0x5617ca52d400 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3c9be00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca534a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca531f00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009c80 tx=0x7f3c90011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4881ea4f70aa4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.755+0000 7f3c99600640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 0x7f3c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.756+0000 7f3c8e000640 1 -- 172.21.15.98:0/4266246381 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004230 con 0x5617ca52d400 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.756+0000 7f3c8e000640 1 -- 172.21.15.98:0/4266246381 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c900043d0 con 0x5617ca52d400 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.756+0000 7f3c8e000640 1 -- 172.21.15.98:0/4266246381 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004df0 con 0x5617ca52d400 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.756+0000 7f3c99600640 1 -- 172.21.15.98:0/1191581316 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c880f0910 con 0x7f3c9c1028a0 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.756+0000 7f3ca1800640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 0x7f3c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 -- 172.21.15.98:0/4266246381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 msgr2=0x5617ca531f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca531f00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009c80 tx=0x7f3c90011810 comp rx=0 tx=0).stop 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 -- 172.21.15.98:0/4266246381 shutdown_connections 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca534a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca531f00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 --2- 172.21.15.98:0/4266246381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca52ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 -- 172.21.15.98:0/4266246381 >> 172.21.15.98:0/4266246381 conn(0x5617ca527f80 msgr2=0x5617ca528380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca1800640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 0x7f3c840a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fd0 tx=0x7f3c940096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 -- 172.21.15.98:0/4266246381 shutdown_connections 2024-03-16T12:22:50.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.757+0000 7f3ca6627740 1 -- 172.21.15.98:0/4266246381 wait complete. 2024-03-16T12:22:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.758+0000 7f3ca6627740 1 -- start start 2024-03-16T12:22:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.758+0000 7f3ca6627740 1 Processor -- start 2024-03-16T12:22:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 0x5617ca537520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca53e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5617ca535520 con 0x5617ca532570 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617ca44ced0 con 0x5617ca52d400 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5617ca5356a0 con 0x5617ca52ca10 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca6627740 10 client.0 _ll_register_callbacks cb 0x5617ca488220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44018/0 (socket says 172.21.15.98:44018) 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca0e00640 1 -- 172.21.15.98:0/406334682 learned_addr learned my addr 172.21.15.98:0/406334682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3c9be00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 0x5617ca537520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.759+0000 7f3ca1800640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca53e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 -- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 msgr2=0x5617ca537520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 0x5617ca537520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 -- 172.21.15.98:0/406334682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 msgr2=0x5617ca53e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca53e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 -- 172.21.15.98:0/406334682 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ca1a53f0 con 0x5617ca52ca10 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3c9be00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 0x5617ca537520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca1800640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca53e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3c90000c00 tx=0x7f3c90019a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79f861ef5c20fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3c82a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90028020 con 0x5617ca52ca10 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c900043d0 con 0x5617ca52ca10 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.760+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c90024420 con 0x5617ca52ca10 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.761+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c900046c0 con 0x5617ca52ca10 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.761+0000 7f3ca6627740 10 client.25771 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.761+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5617ca1650f0 con 0x5617ca52ca10 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.761+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c900043d0 con 0x5617ca52ca10 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c90018070 con 0x5617ca52ca10 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 10 client.25771 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 20 client.25771 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca542b40 0x5617ca544f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 4 client.25771 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617ca4a4be0 con 0x5617ca542b40 2024-03-16T12:22:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3c9be00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca542b40 0x5617ca544f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3c9be00640 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca542b40 0x5617ca544f90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8977c6e8876ad142 server_cookie=71d6b4b14688ccd3 in_seq=0 out_seq=0 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.762+0000 7f3c82a00640 10 client.25771 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3c82a00640 1 -- 172.21.15.98:0/406334682 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617ca4a4be0 con 0x5617ca542b40 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3c82a00640 10 client.25771 handle_command_reply: tid=0 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 client.25771 shutdown 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 msgr2=0x5617ca536fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3c90000c00 tx=0x7f3c90019a50 comp rx=0 tx=0).stop 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 shutdown_connections 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca53e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52d400 0x5617ca537520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca542b40 0x5617ca544f90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 --2- 172.21.15.98:0/406334682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52ca10 0x5617ca536fe0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 >> 172.21.15.98:0/406334682 conn(0x5617ca527f80 msgr2=0x5617ca527110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.763+0000 7f3c82a00640 0 client.25771 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.764+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 shutdown_connections 2024-03-16T12:22:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.764+0000 7f3ca6627740 1 -- 172.21.15.98:0/406334682 wait complete. 2024-03-16T12:22:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.764+0000 7f3ca6627740 20 client.25771 trim_cache size 0 max 16384 2024-03-16T12:22:50.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 Processor -- start 2024-03-16T12:22:50.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 -- start start 2024-03-16T12:22:50.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca526f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca536600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1b3090 con 0x5617ca532570 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1b0960 con 0x5617ca52ca10 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.782+0000 7f3ca6627740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1ae7b0 con 0x5617ca52d400 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca526f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3ca0e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca526f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:54576/0 (socket says 172.21.15.98:54576) 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3ca0e00640 1 -- 172.21.15.98:0/3770064631 learned_addr learned my addr 172.21.15.98:0/3770064631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 -- 172.21.15.98:0/3770064631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 msgr2=0x5617ca526f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca526f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 -- 172.21.15.98:0/3770064631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 msgr2=0x5617ca536600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca536600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 -- 172.21.15.98:0/3770064631 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617ca522560 con 0x5617ca52d400 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c9be00640 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53e520 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3c880bcf30 tx=0x7f3c880f6530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1441ecf197797db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c8d600640 1 -- 172.21.15.98:0/3770064631 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8804b040 con 0x5617ca52d400 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.783+0000 7f3c8d600640 1 -- 172.21.15.98:0/3770064631 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c880bcfa0 con 0x5617ca52d400 2024-03-16T12:22:50.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3c8d600640 1 -- 172.21.15.98:0/3770064631 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c880ee160 con 0x5617ca52d400 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 -- 172.21.15.98:0/3770064631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 msgr2=0x5617ca53e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53e520 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3c880bcf30 tx=0x7f3c880f6530 comp rx=0 tx=0).stop 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 -- 172.21.15.98:0/3770064631 shutdown_connections 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca532570 0x5617ca536600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53e520 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 --2- 172.21.15.98:0/3770064631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca52ca10 0x5617ca526f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 -- 172.21.15.98:0/3770064631 >> 172.21.15.98:0/3770064631 conn(0x5617ca527f80 msgr2=0x5617ca526880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 -- 172.21.15.98:0/3770064631 shutdown_connections 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.784+0000 7f3ca6627740 1 -- 172.21.15.98:0/3770064631 wait complete. 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 -- start start 2024-03-16T12:22:50.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 Processor -- start 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca52ca10 0x5617ca53ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca532570 0x5617ca53cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1aee20 con 0x5617ca52ca10 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1b0290 con 0x5617ca532570 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.785+0000 7f3ca6627740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1af4f0 con 0x5617ca52d400 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3ca6627740 10 client.0 _ll_register_callbacks cb 0x7f3c880f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44024/0 (socket says 172.21.15.98:44024) 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 -- 172.21.15.98:0/339327062 learned_addr learned my addr 172.21.15.98:0/339327062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca52ca10 0x5617ca53ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3ca1800640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca532570 0x5617ca53cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 -- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca532570 msgr2=0x5617ca53cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca532570 0x5617ca53cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 -- 172.21.15.98:0/339327062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca52ca10 msgr2=0x5617ca53ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:22:50.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca52ca10 0x5617ca53ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 -- 172.21.15.98:0/339327062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ca53d590 con 0x5617ca52d400 2024-03-16T12:22:50.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c9be00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009900 tx=0x7f3c88005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333a27c3912940db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:22:50.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.786+0000 7f3c82a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8804b070 con 0x5617ca52d400 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c880035f0 con 0x5617ca52d400 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c880bcfa0 con 0x5617ca52d400 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c880bc900 con 0x5617ca52d400 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3ca6627740 10 client.25777 fetch_fsmap learned FSMap version 24 2024-03-16T12:22:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5617ca53e910 con 0x5617ca52d400 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c880bcfa0 con 0x5617ca52d400 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.787+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c880bc4f0 con 0x5617ca52d400 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 10 client.25777 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 10 client.25777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 20 client.25777 populate_metadata read hostname 'smithi098' 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca5a9590 0x5617ca5ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 4 client.25777 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5617ca49e730 con 0x5617ca5a9590 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca5a9590 0x5617ca5ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3ca0e00640 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca5a9590 0x5617ca5ab9b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c26272f691669d2 server_cookie=c3187ec56df6cc8d in_seq=0 out_seq=0 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3c82a00640 10 client.25777 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3c82a00640 1 -- 172.21.15.98:0/339327062 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5617ca49e730 con 0x5617ca5a9590 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.788+0000 7f3c82a00640 10 client.25777 handle_command_reply: tid=0 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 client.25777 shutdown 2024-03-16T12:22:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 msgr2=0x5617ca53bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009900 tx=0x7f3c88005fd0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 shutdown_connections 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617ca532570 0x5617ca53cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5617ca5a9590 0x5617ca5ab9b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5617ca52d400 0x5617ca53bf90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 --2- 172.21.15.98:0/339327062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5617ca52ca10 0x5617ca53ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 >> 172.21.15.98:0/339327062 conn(0x5617ca527f80 msgr2=0x5617ca52b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3c82a00640 0 client.25777 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 shutdown_connections 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 1 -- 172.21.15.98:0/339327062 wait complete. 2024-03-16T12:22:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.789+0000 7f3ca6627740 20 client.25777 trim_cache size 0 max 16384 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "00698bdf-8251-490e-9205-9a45c75a4b83": { 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "00698bdf-8251-490e-9205-9a45c75a4b83", 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:22:50.774 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 msgr2=0x7f3c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 0x7f3c840a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fd0 tx=0x7f3c940096a0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 msgr2=0x7f3c9c19e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c19e4f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009230 tx=0x7f3c8802c6c0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 shutdown_connections 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f3c840a2800 0x7f3c840a4cc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c9c105250 0x7f3c9c1a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c9c1028a0 0x7f3c9c19e4f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 --2- 172.21.15.98:0/1191581316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c9c0fff30 0x7f3c9c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:22:50.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 >> 172.21.15.98:0/1191581316 conn(0x7f3c9c0fbb50 msgr2=0x7f3c9c1008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:22:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 shutdown_connections 2024-03-16T12:22:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:22:50.790+0000 7f3ca2200640 1 -- 172.21.15.98:0/1191581316 wait complete. 2024-03-16T12:22:50.788 INFO:tasks.cephfs.filesystem:scrub status for tag:00698bdf-8251-490e-9205-9a45c75a4b83 - {'path': '/', 'tag': '00698bdf-8251-490e-9205-9a45c75a4b83', 'options': 'recursive,force'} 2024-03-16T12:22:52.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:51 smithi098 ceph-mon[31633]: scrub queued for path: / 2024-03-16T12:22:52.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:51 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:51 smithi098 ceph-mon[31633]: scrub summary: active paths [/] 2024-03-16T12:22:52.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:51 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:51 smithi098 ceph-mon[31633]: pgmap v445: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:22:52.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:51 smithi153 ceph-mon[36901]: scrub queued for path: / 2024-03-16T12:22:52.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:51 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:51 smithi153 ceph-mon[36901]: scrub summary: active paths [/] 2024-03-16T12:22:52.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:51 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:51 smithi153 ceph-mon[36901]: pgmap v445: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:22:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:51 smithi173 ceph-mon[36632]: scrub queued for path: / 2024-03-16T12:22:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:51 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:51 smithi173 ceph-mon[36632]: scrub summary: active paths [/] 2024-03-16T12:22:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:51 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:22:52.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:51 smithi173 ceph-mon[36632]: pgmap v445: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:22:54.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:53 smithi098 ceph-mon[31633]: pgmap v446: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:54.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:53 smithi153 ceph-mon[36901]: pgmap v446: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:54.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:53 smithi173 ceph-mon[36632]: pgmap v446: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:22:55.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:55 smithi098 ceph-mon[31633]: scrub summary: idle 2024-03-16T12:22:55.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:55 smithi153 ceph-mon[36901]: scrub summary: idle 2024-03-16T12:22:55.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:55 smithi173 ceph-mon[36632]: scrub summary: idle 2024-03-16T12:22:56.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:56 smithi098 ceph-mon[31633]: pgmap v447: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:56.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:56 smithi153 ceph-mon[36901]: pgmap v447: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:56.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:56 smithi173 ceph-mon[36632]: pgmap v447: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:22:58.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:22:58 smithi098 ceph-mon[31633]: pgmap v448: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:22:58.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:22:58 smithi153 ceph-mon[36901]: pgmap v448: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:22:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:22:58 smithi173 ceph-mon[36632]: pgmap v448: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:23:00.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:00 smithi098 ceph-mon[31633]: pgmap v449: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T12:23:00.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:00 smithi153 ceph-mon[36901]: pgmap v449: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T12:23:00.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:00 smithi173 ceph-mon[36632]: pgmap v449: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T12:23:02.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:02 smithi153 ceph-mon[36901]: pgmap v450: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T12:23:02.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:02 smithi173 ceph-mon[36632]: pgmap v450: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T12:23:02.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:02 smithi098 ceph-mon[31633]: pgmap v450: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T12:23:04.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:04 smithi153 ceph-mon[36901]: pgmap v451: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:23:04.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:04 smithi173 ceph-mon[36632]: pgmap v451: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:23:04.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:04 smithi098 ceph-mon[31633]: pgmap v451: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T12:23:06.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:06 smithi153 ceph-mon[36901]: pgmap v452: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-16T12:23:06.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:06 smithi173 ceph-mon[36632]: pgmap v452: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-16T12:23:06.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:06 smithi098 ceph-mon[31633]: pgmap v452: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-16T12:23:08.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:08 smithi173 ceph-mon[36632]: pgmap v453: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:23:08.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:08 smithi098 ceph-mon[31633]: pgmap v453: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:23:08.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:08 smithi153 ceph-mon[36901]: pgmap v453: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-16T12:23:08.610 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (7790/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'. 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T12:23:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:09.307 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 3d78f5ba07b Merge PR #54581 into wip-batrick-testing-20240314.183243 2024-03-16T12:23:09.316 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T12:23:09.380 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T12:23:09.387 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T12:23:09.387 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T12:23:10.281 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T12:23:10.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:10 smithi173 ceph-mon[36632]: pgmap v454: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:23:10.413 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T12:23:10.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:10 smithi098 ceph-mon[31633]: pgmap v454: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:23:10.498 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T12:23:10.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:10 smithi153 ceph-mon[36901]: pgmap v454: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:23:10.504 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T12:23:10.505 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T12:23:10.586 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T12:23:10.596 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:23:10.596 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T12:23:10.645 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-03-16T12:23:10.646 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-03-16T12:23:10.646 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-16T12:23:10.762 INFO:tasks.workunit.client.0.smithi098.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-03-16T12:23:10.811 INFO:tasks.workunit.client.0.smithi098.stderr:--2024-03-16 12:23:10-- http://download.ceph.com/qa/fsync-tester.c 2024-03-16T12:23:10.813 INFO:tasks.workunit.client.0.smithi098.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-16T12:23:10.849 INFO:tasks.workunit.client.0.smithi098.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-16T12:23:10.886 INFO:tasks.workunit.client.0.smithi098.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-16T12:23:10.886 INFO:tasks.workunit.client.0.smithi098.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-03-16T12:23:10.889 INFO:tasks.workunit.client.0.smithi098.stderr:Saving to: ‘fsync-tester.c’ 2024-03-16T12:23:10.896 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:10.896 INFO:tasks.workunit.client.0.smithi098.stderr: 0K .. 100% 74.7M=0s 2024-03-16T12:23:10.896 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:10.896 INFO:tasks.workunit.client.0.smithi098.stderr:2024-03-16 12:23:10 (74.7 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-03-16T12:23:10.896 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-03-16T12:23:10.898 INFO:tasks.workunit.client.0.smithi098.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-03-16T12:23:11.028 INFO:tasks.workunit.client.0.smithi098.stderr:+ ./fsync-tester 2024-03-16T12:23:12.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:12 smithi173 ceph-mon[36632]: pgmap v455: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:23:12.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:12 smithi098 ceph-mon[31633]: pgmap v455: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:23:12.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:12 smithi153 ceph-mon[36901]: pgmap v455: 129 pgs: 129 active+clean; 658 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:23:14.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:14 smithi173 ceph-mon[36632]: pgmap v456: 129 pgs: 129 active+clean; 695 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:14.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:14 smithi098 ceph-mon[31633]: pgmap v456: 129 pgs: 129 active+clean; 695 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:14.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:14 smithi153 ceph-mon[36901]: pgmap v456: 129 pgs: 129 active+clean; 695 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:16.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:16 smithi098 ceph-mon[31633]: pgmap v457: 129 pgs: 129 active+clean; 695 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:16.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:16 smithi173 ceph-mon[36632]: pgmap v457: 129 pgs: 129 active+clean; 695 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:16.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:16 smithi153 ceph-mon[36901]: pgmap v457: 129 pgs: 129 active+clean; 695 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-03-16T12:23:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:18 smithi173 ceph-mon[36632]: pgmap v458: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:18.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:18 smithi098 ceph-mon[31633]: pgmap v458: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:18 smithi153 ceph-mon[36901]: pgmap v458: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:20 smithi173 ceph-mon[36632]: pgmap v459: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:20 smithi098 ceph-mon[31633]: pgmap v459: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:20 smithi153 ceph-mon[36901]: pgmap v459: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 434 KiB/s wr, 8 op/s 2024-03-16T12:23:20.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:23:20.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 -- 172.21.15.98:0/995067507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec0761a0 msgr2=0x7f11ec0765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 --2- 172.21.15.98:0/995067507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec0761a0 0x7f11ec0765c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009900 tx=0x7f11e002bc40 comp rx=0 tx=0).stop 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 -- 172.21.15.98:0/995067507 shutdown_connections 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 --2- 172.21.15.98:0/995067507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec077d80 0x7f11ec078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 --2- 172.21.15.98:0/995067507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 0x7f11ec077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:20.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 --2- 172.21.15.98:0/995067507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec0761a0 0x7f11ec0765c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:20.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 -- 172.21.15.98:0/995067507 >> 172.21.15.98:0/995067507 conn(0x7f11ec0fef90 msgr2=0x7f11ec1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:20.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 -- 172.21.15.98:0/995067507 shutdown_connections 2024-03-16T12:23:20.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.016+0000 7f11f3e00640 1 -- 172.21.15.98:0/995067507 wait complete. 2024-03-16T12:23:20.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.017+0000 7f11f3e00640 1 Processor -- start 2024-03-16T12:23:20.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.017+0000 7f11f3e00640 1 -- start start 2024-03-16T12:23:20.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:20.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 0x7f11ec19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:20.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 0x7f11ec1a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113ab0 con 0x7f11ec0773e0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113930 con 0x7f11ec0761a0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec113c30 con 0x7f11ec077d80 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 0x7f11ec1a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44270/0 (socket says 172.21.15.98:44270) 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f1400640 1 -- 172.21.15.98:0/2025734700 learned_addr learned my addr 172.21.15.98:0/2025734700 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f0a00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 0x7f11ec19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.018+0000 7f11f1400640 1 -- 172.21.15.98:0/2025734700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 msgr2=0x7f11ec1a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1400640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 0x7f11ec1a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1400640 1 -- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 msgr2=0x7f11ec19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1400640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 0x7f11ec19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1400640 1 -- 172.21.15.98:0/2025734700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec1a5d70 con 0x7f11ec0761a0 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 0x7f11ec1a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f1400640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f11e002c8f0 tx=0x7f11e002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11ea000640 1 -- 172.21.15.98:0/2025734700 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0041070 con 0x7f11ec0761a0 2024-03-16T12:23:21.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11ea000640 1 -- 172.21.15.98:0/2025734700 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0004450 con 0x7f11ec0761a0 2024-03-16T12:23:21.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec1a6000 con 0x7f11ec0761a0 2024-03-16T12:23:21.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11ea000640 1 -- 172.21.15.98:0/2025734700 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0003df0 con 0x7f11ec0761a0 2024-03-16T12:23:21.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.019+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1a64f0 con 0x7f11ec0761a0 2024-03-16T12:23:21.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.022+0000 7f11f6ba8740 1 Processor -- start 2024-03-16T12:23:21.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 -- start start 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a769f200 0x55c7a76a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 0x55c7a76a6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7a76031f0 con 0x55c7a769e810 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c7a75aba80 con 0x55c7a76a4370 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7a75ff290 con 0x55c7a769f200 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56030/0 (socket says 172.21.15.98:56030) 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f1e00640 1 -- 172.21.15.98:0/1857722863 learned_addr learned my addr 172.21.15.98:0/1857722863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.023+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 0x55c7a76a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 -- 172.21.15.98:0/1857722863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a769f200 msgr2=0x55c7a76a3d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a769f200 0x55c7a76a3d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 -- 172.21.15.98:0/1857722863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 msgr2=0x55c7a76a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 0x55c7a76a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 -- 172.21.15.98:0/1857722863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a73e17d0 con 0x55c7a769e810 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 0x55c7a76a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11ea000640 1 -- 172.21.15.98:0/2025734700 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f11e0044070 con 0x7f11ec0761a0 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11f1e00640 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011810 tx=0x7f11e4011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5317fc724b1eec6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11de600640 1 -- 172.21.15.98:0/1857722863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4011310 con 0x55c7a769e810 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11de600640 1 -- 172.21.15.98:0/1857722863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e400f9a0 con 0x55c7a769e810 2024-03-16T12:23:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.024+0000 7f11de600640 1 -- 172.21.15.98:0/1857722863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e40056c0 con 0x55c7a769e810 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.025+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1857722863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 msgr2=0x55c7a769ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.025+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011810 tx=0x7f11e4011cd0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.025+0000 7f11ea000640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 0x7f11d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.025+0000 7f11ea000640 1 -- 172.21.15.98:0/2025734700 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11e00f0920 con 0x7f11ec0761a0 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1857722863 shutdown_connections 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f1e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 0x7f11d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a76a4370 0x55c7a76a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a769f200 0x55c7a76a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1857722863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a769ec30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1857722863 >> 172.21.15.98:0/1857722863 conn(0x55c7a7699d80 msgr2=0x55c7a769a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1857722863 shutdown_connections 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1857722863 wait complete. 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f1e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 0x7f11d40a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.026+0000 7f11f6ba8740 1 -- start start 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 Processor -- start 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7a76a7320 con 0x55c7a769e810 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c7a75beed0 con 0x55c7a769f200 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7a76a74a0 con 0x55c7a76a4370 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f6ba8740 10 client.0 _ll_register_callbacks cb 0x55c7a75fa220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:21.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.027+0000 7f11f1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44294/0 (socket says 172.21.15.98:44294) 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 -- 172.21.15.98:0/1911563993 learned_addr learned my addr 172.21.15.98:0/1911563993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 -- 172.21.15.98:0/1911563993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 msgr2=0x55c7a76b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 -- 172.21.15.98:0/1911563993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 msgr2=0x55c7a76a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 -- 172.21.15.98:0/1911563993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a73173f0 con 0x55c7a769f200 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f1400640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f11d8012700 tx=0x7f11d8012b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9746127695ebb76 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d800f640 con 0x55c7a769f200 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11d800f7e0 con 0x55c7a769f200 2024-03-16T12:23:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.028+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e400f9a0 con 0x55c7a769f200 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f11d800f9e0 con 0x55c7a769f200 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 10 client.25799 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c7a72d70f0 con 0x55c7a769f200 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e400f9a0 con 0x55c7a769f200 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f11d8010ce0 con 0x55c7a769f200 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 10 client.25799 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 10 client.25799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 20 client.25799 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a76b4c70 0x55c7a76b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 4 client.25799 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.029+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7a7616be0 con 0x55c7a76b4c70 2024-03-16T12:23:21.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.030+0000 7f11f1400640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a76b4c70 0x55c7a76b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.030+0000 7f11f1400640 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a76b4c70 0x55c7a76b70b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16f96b9b462a06ab server_cookie=cac384c52978906b in_seq=0 out_seq=0 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.030+0000 7f11d2a00640 10 client.25799 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11d2a00640 1 -- 172.21.15.98:0/1911563993 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7a7616be0 con 0x55c7a76b4c70 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11d2a00640 10 client.25799 handle_command_reply: tid=0 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 client.25799 shutdown 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 msgr2=0x55c7a76a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f11d8012700 tx=0x7f11d8012b00 comp rx=0 tx=0).stop 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 shutdown_connections 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a76b4c70 0x55c7a76b70b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76a9360 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11d2a00640 0 client.25799 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1911563993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.031+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 >> 172.21.15.98:0/1911563993 conn(0x55c7a7699d80 msgr2=0x55c7a7698ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.032+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 shutdown_connections 2024-03-16T12:23:21.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.032+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1911563993 wait complete. 2024-03-16T12:23:21.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.032+0000 7f11f6ba8740 20 client.25799 trim_cache size 0 max 16384 2024-03-16T12:23:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 Processor -- start 2024-03-16T12:23:21.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 -- start start 2024-03-16T12:23:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769f200 0x55c7a76b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a7698b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b3290 con 0x55c7a769f200 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b0b60 con 0x55c7a769e810 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1ae9b0 con 0x55c7a76a4370 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.051+0000 7f11f1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f0a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769f200 0x55c7a76b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44306/0 (socket says 172.21.15.98:44306) 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 -- 172.21.15.98:0/3482468382 learned_addr learned my addr 172.21.15.98:0/3482468382 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1e00640 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a7698b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 -- 172.21.15.98:0/3482468382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 msgr2=0x55c7a7698b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a7698b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 -- 172.21.15.98:0/3482468382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769f200 msgr2=0x55c7a76b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769f200 0x55c7a76b0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 -- 172.21.15.98:0/3482468382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a769fa90 con 0x55c7a769e810 2024-03-16T12:23:21.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11f1400640 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f11e00b9360 tx=0x7f11e0003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=332eb45eb1cd369d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11ddc00640 1 -- 172.21.15.98:0/3482468382 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e004b040 con 0x55c7a769e810 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11ddc00640 1 -- 172.21.15.98:0/3482468382 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e00bcf00 con 0x55c7a769e810 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.052+0000 7f11ddc00640 1 -- 172.21.15.98:0/3482468382 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0004c60 con 0x55c7a769e810 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.053+0000 7f11f6ba8740 1 -- 172.21.15.98:0/3482468382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 msgr2=0x55c7a7698630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.053+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f11e00b9360 tx=0x7f11e0003f90 comp rx=0 tx=0).stop 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 -- 172.21.15.98:0/3482468382 shutdown_connections 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a7698b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769f200 0x55c7a76b0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/3482468382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769e810 0x55c7a7698630 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 -- 172.21.15.98:0/3482468382 >> 172.21.15.98:0/3482468382 conn(0x55c7a7699d80 msgr2=0x55c7a76a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 -- 172.21.15.98:0/3482468382 shutdown_connections 2024-03-16T12:23:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 -- 172.21.15.98:0/3482468382 wait complete. 2024-03-16T12:23:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 -- start start 2024-03-16T12:23:21.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.054+0000 7f11f6ba8740 1 Processor -- start 2024-03-16T12:23:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76adc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af020 con 0x55c7a769e810 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b0490 con 0x55c7a769f200 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af6f0 con 0x55c7a76a4370 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f6ba8740 10 client.0 _ll_register_callbacks cb 0x7f11d8009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44310/0 (socket says 172.21.15.98:44310) 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 -- 172.21.15.98:0/1125273171 learned_addr learned my addr 172.21.15.98:0/1125273171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f1e00640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f1400640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 -- 172.21.15.98:0/1125273171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 msgr2=0x55c7a76af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 -- 172.21.15.98:0/1125273171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 msgr2=0x55c7a76adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.055+0000 7f11f0a00640 1 -- 172.21.15.98:0/1125273171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a76af670 con 0x55c7a769f200 2024-03-16T12:23:21.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11f0a00640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55c7a76aa300 tx=0x7f11d800fdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd86b7ef3917207 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8009dd0 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e00bcf00 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11d80050b0 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f11d8037050 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11f6ba8740 10 client.25805 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c7a771b080 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.056+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e00bcf00 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f11d8032030 con 0x55c7a769f200 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 10 client.25805 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 20 client.25805 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a771ba50 0x55c7a771de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 4 client.25805 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7a7610730 con 0x55c7a771ba50 2024-03-16T12:23:21.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f1400640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a771ba50 0x55c7a771de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11f1400640 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a771ba50 0x55c7a771de70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b98832b753555b0b server_cookie=c28ee022da01c987 in_seq=0 out_seq=0 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.057+0000 7f11d2a00640 10 client.25805 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11d2a00640 1 -- 172.21.15.98:0/1125273171 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c7a7610730 con 0x55c7a771ba50 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11d2a00640 10 client.25805 handle_command_reply: tid=0 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 client.25805 shutdown 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 msgr2=0x55c7a76ae180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55c7a76aa300 tx=0x7f11d800fdb0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 shutdown_connections 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55c7a771ba50 0x55c7a771de70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7a76a4370 0x55c7a76af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c7a769f200 0x55c7a76ae180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 --2- 172.21.15.98:0/1125273171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7a769e810 0x55c7a76adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11d2a00640 0 client.25805 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 >> 172.21.15.98:0/1125273171 conn(0x55c7a7699d80 msgr2=0x55c7a769cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 shutdown_connections 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 1 -- 172.21.15.98:0/1125273171 wait complete. 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.058+0000 7f11f6ba8740 20 client.25805 trim_cache size 0 max 16384 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-03-16T12:23:21.042 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 msgr2=0x7f11d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 0x7f11d40a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f11e4005fd0 tx=0x7f11e4000f40 comp rx=0 tx=0).stop 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 msgr2=0x7f11ec19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f11e002c8f0 tx=0x7f11e002c660 comp rx=0 tx=0).stop 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 shutdown_connections 2024-03-16T12:23:21.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f11d40a2800 0x7f11d40a4cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11ec077d80 0x7f11ec1a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11ec0773e0 0x7f11ec19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 --2- 172.21.15.98:0/2025734700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11ec0761a0 0x7f11ec19e150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 >> 172.21.15.98:0/2025734700 conn(0x7f11ec0fef90 msgr2=0x7f11ec100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 shutdown_connections 2024-03-16T12:23:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.059+0000 7f11f3e00640 1 -- 172.21.15.98:0/2025734700 wait complete. 2024-03-16T12:23:21.055 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T12:23:21.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:23:21.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.273+0000 7fc1d9a00640 1 -- 172.21.15.98:0/4093942140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41029e0 msgr2=0x7fc1d4102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.273+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/4093942140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41029e0 0x7fc1d4102e60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c002bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:21.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 -- 172.21.15.98:0/4093942140 shutdown_connections 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/4093942140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/4093942140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41029e0 0x7fc1d4102e60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/4093942140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41017f0 0x7fc1d4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 -- 172.21.15.98:0/4093942140 >> 172.21.15.98:0/4093942140 conn(0x7fc1d40fcd80 msgr2=0x7fc1d40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 -- 172.21.15.98:0/4093942140 shutdown_connections 2024-03-16T12:23:21.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 -- 172.21.15.98:0/4093942140 wait complete. 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.274+0000 7fc1d9a00640 1 Processor -- start 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 -- start start 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41029e0 0x7fc1d419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d41a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d4c0 con 0x7fc1d41029e0 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d340 con 0x7fc1d41017f0 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d9a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410d640 con 0x7fc1d41033a0 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41029e0 0x7fc1d419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d41a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44326/0 (socket says 172.21.15.98:44326) 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.275+0000 7fc1d2a00640 1 -- 172.21.15.98:0/1401960386 learned_addr learned my addr 172.21.15.98:0/1401960386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 -- 172.21.15.98:0/1401960386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 msgr2=0x7fc1d41a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d41a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 -- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41029e0 msgr2=0x7fc1d419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41029e0 0x7fc1d419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 -- 172.21.15.98:0/1401960386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d41a1930 con 0x7fc1d41017f0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d3400640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d2a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9c0 tx=0x7fc1c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400cd80 con 0x7fc1d41017f0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c40035f0 con 0x7fc1d41017f0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4010690 con 0x7fc1d41017f0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d41a1c20 con 0x7fc1d41017f0 2024-03-16T12:23:21.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.276+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d41a21c0 con 0x7fc1d41017f0 2024-03-16T12:23:21.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.277+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d41a2390 con 0x7fc1d41017f0 2024-03-16T12:23:21.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.279+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc1c40037a0 con 0x7fc1d41017f0 2024-03-16T12:23:21.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.280+0000 7fc1cb400640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 0x7fc1b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.280+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc1c40cf1c0 con 0x7fc1d41017f0 2024-03-16T12:23:21.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.281+0000 7fc1d2000640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 0x7fc1b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.281+0000 7fc1d2000640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 0x7fc1b80a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc1c002c240 tx=0x7fc1c002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.289+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1c409bea0 con 0x7fc1d41017f0 2024-03-16T12:23:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.526+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1d41a2560 con 0x7fc1d41017f0 2024-03-16T12:23:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.527+0000 7fc1cb400640 1 -- 172.21.15.98:0/1401960386 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7fc1c409b7a0 con 0x7fc1d41017f0 2024-03-16T12:23:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:23:21.507 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:23:21.507 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 msgr2=0x7fc1b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 0x7fc1b80a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc1c002c240 tx=0x7fc1c002df90 comp rx=0 tx=0).stop 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 msgr2=0x7fc1d4199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9c0 tx=0x7fc1c400ee80 comp rx=0 tx=0).stop 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 shutdown_connections 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fc1b80a2880 0x7fc1b80a4d40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1d41033a0 0x7fc1d41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d41029e0 0x7fc1d419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 --2- 172.21.15.98:0/1401960386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1d41017f0 0x7fc1d4199d20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.529+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 >> 172.21.15.98:0/1401960386 conn(0x7fc1d40fcd80 msgr2=0x7fc1d4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.530+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 shutdown_connections 2024-03-16T12:23:21.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.530+0000 7fc1d9a00640 1 -- 172.21.15.98:0/1401960386 wait complete. 2024-03-16T12:23:21.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 -- 172.21.15.98:0/3054645260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 msgr2=0x7fdff8109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 --2- 172.21.15.98:0/3054645260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff8109930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009ca0 tx=0x7fdff002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 -- 172.21.15.98:0/3054645260 shutdown_connections 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 --2- 172.21.15.98:0/3054645260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff8109930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 --2- 172.21.15.98:0/3054645260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 --2- 172.21.15.98:0/3054645260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff81046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 -- 172.21.15.98:0/3054645260 >> 172.21.15.98:0/3054645260 conn(0x7fdff80fbc50 msgr2=0x7fdff80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 -- 172.21.15.98:0/3054645260 shutdown_connections 2024-03-16T12:23:21.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.749+0000 7fe000000640 1 -- 172.21.15.98:0/3054645260 wait complete. 2024-03-16T12:23:21.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.750+0000 7fe000000640 1 Processor -- start 2024-03-16T12:23:21.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 -- start start 2024-03-16T12:23:21.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff810d480 con 0x7fdff8104c10 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdff810d300 con 0x7fdff81022e0 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fe000000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff810d600 con 0x7fdff8107540 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fdffd600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fdffd600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44342/0 (socket says 172.21.15.98:44342) 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fdffd600640 1 -- 172.21.15.98:0/3120012032 learned_addr learned my addr 172.21.15.98:0/3120012032 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fdffe000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.751+0000 7fdffcc00640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 -- 172.21.15.98:0/3120012032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 msgr2=0x7fdff819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 -- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 msgr2=0x7fdff8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 -- 172.21.15.98:0/3120012032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff819f6f0 con 0x7fdff81022e0 2024-03-16T12:23:21.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffcc00640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdffd600640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e9f0 tx=0x7fdfec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdff6000640 1 -- 172.21.15.98:0/3120012032 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00cd80 con 0x7fdff81022e0 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdff6000640 1 -- 172.21.15.98:0/3120012032 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfec005500 con 0x7fdff81022e0 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff819f9e0 con 0x7fdff81022e0 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fdff6000640 1 -- 172.21.15.98:0/3120012032 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec010690 con 0x7fdff81022e0 2024-03-16T12:23:21.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.752+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff819fee0 con 0x7fdff81022e0 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.755+0000 7fe002d8d740 1 Processor -- start 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.755+0000 7fe002d8d740 1 -- start start 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdff6000640 1 -- 172.21.15.98:0/3120012032 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdfec0046c0 con 0x7fdff81022e0 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5211c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f521830 0x55ba7f523cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f4df3b0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f47c450 con 0x55ba7f521830 2024-03-16T12:23:21.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fe002d8d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f424cb0 con 0x55ba7f51bcd0 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffd600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49858/0 (socket says 172.21.15.98:49858) 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 -- 172.21.15.98:0/3101635907 learned_addr learned my addr 172.21.15.98:0/3101635907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f521830 0x55ba7f523cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 -- 172.21.15.98:0/3101635907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f521830 msgr2=0x55ba7f523cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f521830 0x55ba7f523cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 -- 172.21.15.98:0/3101635907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 msgr2=0x55ba7f5211c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.756+0000 7fdffe000640 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5211c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdffe000640 1 -- 172.21.15.98:0/3101635907 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba7f4461f0 con 0x55ba7f51bcd0 2024-03-16T12:23:21.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdffe000640 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009ca0 tx=0x7fdff00323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53bdba37cb23d8af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdff6000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 0x7fdfe00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdfea600640 1 -- 172.21.15.98:0/3101635907 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0043070 con 0x55ba7f51bcd0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdfea600640 1 -- 172.21.15.98:0/3101635907 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff0032c90 con 0x55ba7f51bcd0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdfea600640 1 -- 172.21.15.98:0/3101635907 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0039040 con 0x55ba7f51bcd0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdff6000640 1 -- 172.21.15.98:0/3120012032 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdfec0cf260 con 0x7fdff81022e0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.757+0000 7fdffd600640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 0x7fdfe00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 -- 172.21.15.98:0/3101635907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 msgr2=0x55ba7f51c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009ca0 tx=0x7fdff00323c0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fdffd600640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 0x7fdfe00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0ccd70 tx=0x7fdfec0d2d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 -- 172.21.15.98:0/3101635907 shutdown_connections 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f521830 0x55ba7f523cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5211c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3101635907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51bcd0 0x55ba7f51c0f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 -- 172.21.15.98:0/3101635907 >> 172.21.15.98:0/3101635907 conn(0x55ba7f512e30 msgr2=0x55ba7f513230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 -- 172.21.15.98:0/3101635907 shutdown_connections 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.758+0000 7fe002d8d740 1 -- 172.21.15.98:0/3101635907 wait complete. 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 -- start start 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 Processor -- start 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 0x55ba7f525f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f521830 0x55ba7f52d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f438110 con 0x55ba7f51bcd0 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f4784f0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ba7f524780 con 0x55ba7f521830 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fe002d8d740 10 client.0 _ll_register_callbacks cb 0x55ba7f425b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:21.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44362/0 (socket says 172.21.15.98:44362) 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.759+0000 7fdffcc00640 1 -- 172.21.15.98:0/3331137691 learned_addr learned my addr 172.21.15.98:0/3331137691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffe000640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 0x55ba7f525f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffe000640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f521830 0x55ba7f52d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 -- 172.21.15.98:0/3331137691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f521830 msgr2=0x55ba7f52d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f521830 0x55ba7f52d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 -- 172.21.15.98:0/3331137691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 msgr2=0x55ba7f525f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 0x55ba7f525f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 -- 172.21.15.98:0/3331137691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba7f1903f0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffe000640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 0x55ba7f525f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdffcc00640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400b900 tx=0x7fdfe400bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57cd540b735d8e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdfdea00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe401b640 con 0x55ba7f51c6c0 2024-03-16T12:23:21.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff0032c90 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.760+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdfe401b7e0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdfe4024050 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 10 client.25823 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ba7f25a7d0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff0032c90 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdfe4019020 con 0x55ba7f51c6c0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 10 client.25823 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 10 client.25823 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 20 client.25823 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f531de0 0x55ba7f534240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 4 client.25823 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba7f489990 con 0x55ba7f531de0 2024-03-16T12:23:21.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.761+0000 7fdffe000640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f531de0 0x55ba7f534240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.762+0000 7fdffe000640 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f531de0 0x55ba7f534240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3ae49245fd23bac server_cookie=8212ace527e17a29 in_seq=0 out_seq=0 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.762+0000 7fdfdea00640 10 client.25823 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.762+0000 7fdfdea00640 1 -- 172.21.15.98:0/3331137691 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ba7f489990 con 0x55ba7f531de0 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.762+0000 7fdfdea00640 10 client.25823 handle_command_reply: tid=0 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 client.25823 shutdown 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 msgr2=0x55ba7f5264c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400b900 tx=0x7fdfe400bdc0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 shutdown_connections 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f531de0 0x55ba7f534240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f521830 0x55ba7f52d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51c6c0 0x55ba7f5264c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 --2- 172.21.15.98:0/3331137691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51bcd0 0x55ba7f525f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fdfdea00640 0 client.25823 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 >> 172.21.15.98:0/3331137691 conn(0x55ba7f512e30 msgr2=0x55ba7f5121f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 shutdown_connections 2024-03-16T12:23:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 1 -- 172.21.15.98:0/3331137691 wait complete. 2024-03-16T12:23:21.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.763+0000 7fe002d8d740 20 client.25823 trim_cache size 0 max 16384 2024-03-16T12:23:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.777+0000 7fe002d8d740 1 Processor -- start 2024-03-16T12:23:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.777+0000 7fe002d8d740 1 -- start start 2024-03-16T12:23:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.777+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f512e30 0x55ba7f512070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.777+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.777+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f538b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fe002d8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81aa990 con 0x55ba7f51c6c0 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fe002d8d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81a8260 con 0x55ba7f51bcd0 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fe002d8d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81a60b0 con 0x55ba7f512e30 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44372/0 (socket says 172.21.15.98:44372) 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 -- 172.21.15.98:0/2040097018 learned_addr learned my addr 172.21.15.98:0/2040097018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffe000640 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f512e30 0x55ba7f512070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffe000640 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f538b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 -- 172.21.15.98:0/2040097018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f512e30 msgr2=0x55ba7f512070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f512e30 0x55ba7f512070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 -- 172.21.15.98:0/2040097018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 msgr2=0x55ba7f538b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f538b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 -- 172.21.15.98:0/2040097018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba7f50ec10 con 0x55ba7f51bcd0 2024-03-16T12:23:21.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.778+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4006010 tx=0x7fdfe4011e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98e6d719d2cee3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.779+0000 7fdfe9c00640 1 -- 172.21.15.98:0/2040097018 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe402c030 con 0x55ba7f51bcd0 2024-03-16T12:23:21.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.779+0000 7fdfe9c00640 1 -- 172.21.15.98:0/2040097018 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfe4003f60 con 0x55ba7f51bcd0 2024-03-16T12:23:21.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.779+0000 7fdfe9c00640 1 -- 172.21.15.98:0/2040097018 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe401bb40 con 0x55ba7f51bcd0 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- 172.21.15.98:0/2040097018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 msgr2=0x55ba7f52d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4006010 tx=0x7fdfe4011e70 comp rx=0 tx=0).stop 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- 172.21.15.98:0/2040097018 shutdown_connections 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f51c6c0 0x55ba7f538b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52d4c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2040097018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f512e30 0x55ba7f512070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- 172.21.15.98:0/2040097018 >> 172.21.15.98:0/2040097018 conn(0x55ba7f50e460 msgr2=0x55ba7f511960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- 172.21.15.98:0/2040097018 shutdown_connections 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- 172.21.15.98:0/2040097018 wait complete. 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 -- start start 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.780+0000 7fe002d8d740 1 Processor -- start 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f512e30 0x55ba7f52aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 0x55ba7f52c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81a6720 con 0x55ba7f512e30 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81a7b90 con 0x55ba7f51bcd0 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81a6df0 con 0x55ba7f51c6c0 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fe002d8d740 10 client.0 _ll_register_callbacks cb 0x7fdfe4004330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffe000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f512e30 0x55ba7f52aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44384/0 (socket says 172.21.15.98:44384) 2024-03-16T12:23:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 -- 172.21.15.98:0/2327455368 learned_addr learned my addr 172.21.15.98:0/2327455368 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffe000640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 0x55ba7f52c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 -- 172.21.15.98:0/2327455368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 msgr2=0x55ba7f52c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 0x55ba7f52c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 -- 172.21.15.98:0/2327455368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f512e30 msgr2=0x55ba7f52aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f512e30 0x55ba7f52aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffcc00640 1 -- 172.21.15.98:0/2327455368 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba7f52ca10 con 0x55ba7f51bcd0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.781+0000 7fdffe000640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 0x55ba7f52c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdffcc00640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4011c00 tx=0x7fdfe401b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed1bb2bdb67c68a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe401cde0 con 0x55ba7f51bcd0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfe4003f60 con 0x55ba7f51bcd0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdfe4011440 con 0x55ba7f51bcd0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdfe4011640 con 0x55ba7f51bcd0 2024-03-16T12:23:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 10 client.25829 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ba7f598450 con 0x55ba7f51bcd0 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfe4003f60 con 0x55ba7f51bcd0 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdfe4019050 con 0x55ba7f51bcd0 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 10 client.25829 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 10 client.25829 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 20 client.25829 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f521830 0x55ba7f59ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 4 client.25829 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.782+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba7f483ce0 con 0x55ba7f521830 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdffe000640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f521830 0x55ba7f59ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdffe000640 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f521830 0x55ba7f59ad20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39c8f56709ab6358 server_cookie=7a7cef18d678ae1d in_seq=0 out_seq=0 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdfdea00640 10 client.25829 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdfdea00640 1 -- 172.21.15.98:0/2327455368 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba7f483ce0 con 0x55ba7f521830 2024-03-16T12:23:21.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdfdea00640 10 client.25829 handle_command_reply: tid=0 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 client.25829 shutdown 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 msgr2=0x55ba7f52b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4011c00 tx=0x7fdfe401b7b0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 shutdown_connections 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x55ba7f521830 0x55ba7f59ad20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba7f51c6c0 0x55ba7f52c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba7f51bcd0 0x55ba7f52b410 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 --2- 172.21.15.98:0/2327455368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ba7f512e30 0x55ba7f52aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fdfdea00640 0 client.25829 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.783+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 >> 172.21.15.98:0/2327455368 conn(0x55ba7f50e460 msgr2=0x55ba7f5121f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 shutdown_connections 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe002d8d740 1 -- 172.21.15.98:0/2327455368 wait complete. 2024-03-16T12:23:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe002d8d740 20 client.25829 trim_cache size 0 max 16384 2024-03-16T12:23:21.769 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 msgr2=0x7fdfe00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 0x7fdfe00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0ccd70 tx=0x7fdfec0d2d40 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 msgr2=0x7fdff8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e9f0 tx=0x7fdfec00edf0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 shutdown_connections 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fdfe00a2800 0x7fdfe00a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff8107540 0x7fdff819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff8104c10 0x7fdff8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 --2- 172.21.15.98:0/3120012032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdff81022e0 0x7fdff8197ad0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.784+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 >> 172.21.15.98:0/3120012032 conn(0x7fdff80fbc50 msgr2=0x7fdff8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.785+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 shutdown_connections 2024-03-16T12:23:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:21.785+0000 7fe000000640 1 -- 172.21.15.98:0/3120012032 wait complete. 2024-03-16T12:23:21.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T12:23:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.000+0000 7f8432600640 1 -- 172.21.15.98:0/160852665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c103a00 msgr2=0x7f842c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.000+0000 7f8432600640 1 --2- 172.21.15.98:0/160852665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c103a00 0x7f842c103e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f841c009900 tx=0x7f841c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:21.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 -- 172.21.15.98:0/160852665 shutdown_connections 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 --2- 172.21.15.98:0/160852665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c1055b0 0x7f842c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 --2- 172.21.15.98:0/160852665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 --2- 172.21.15.98:0/160852665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c103a00 0x7f842c103e20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 -- 172.21.15.98:0/160852665 >> 172.21.15.98:0/160852665 conn(0x7f842c0fef90 msgr2=0x7f842c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 -- 172.21.15.98:0/160852665 shutdown_connections 2024-03-16T12:23:21.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.001+0000 7f8432600640 1 -- 172.21.15.98:0/160852665 wait complete. 2024-03-16T12:23:21.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 Processor -- start 2024-03-16T12:23:21.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 -- start start 2024-03-16T12:23:21.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 0x7f842c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c1055b0 0x7f842c1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10f670 con 0x7f842c103a00 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10f4f0 con 0x7f842c104bf0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f8432600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10f7f0 con 0x7f842c1055b0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f842aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f842aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44394/0 (socket says 172.21.15.98:44394) 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f842aa00640 1 -- 172.21.15.98:0/493111238 learned_addr learned my addr 172.21.15.98:0/493111238 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f842be00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c1055b0 0x7f842c1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.002+0000 7f842b400640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 0x7f842c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 -- 172.21.15.98:0/493111238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c1055b0 msgr2=0x7f842c1a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c1055b0 0x7f842c1a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 -- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 msgr2=0x7f842c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 0x7f842c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 -- 172.21.15.98:0/493111238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c1a1950 con 0x7f842c104bf0 2024-03-16T12:23:21.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842b400640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 0x7f842c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f842aa00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f841800e9f0 tx=0x7f841800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f8423e00640 1 -- 172.21.15.98:0/493111238 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841800cd80 con 0x7f842c104bf0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842c1a1c40 con 0x7f842c104bf0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f8423e00640 1 -- 172.21.15.98:0/493111238 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8418005500 con 0x7f842c104bf0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f8423e00640 1 -- 172.21.15.98:0/493111238 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418010690 con 0x7f842c104bf0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.003+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f842c1a2190 con 0x7f842c104bf0 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.006+0000 7f84353e5740 1 Processor -- start 2024-03-16T12:23:21.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.006+0000 7f84353e5740 1 -- start start 2024-03-16T12:23:21.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdd3420 0x55ab1cdd58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab1cd953b0 con 0x55ab1cdcd8c0 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ab1cd32450 con 0x55ab1cdd3420 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f84353e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab1ccdacb0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49906/0 (socket says 172.21.15.98:49906) 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56152/0 (socket says 172.21.15.98:56152) 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 -- 172.21.15.98:0/1281682018 learned_addr learned my addr 172.21.15.98:0/1281682018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f8423e00640 1 -- 172.21.15.98:0/493111238 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f84180046c0 con 0x7f842c104bf0 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 -- 172.21.15.98:0/1281682018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdd3420 msgr2=0x55ab1cdd58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdd3420 0x55ab1cdd58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 -- 172.21.15.98:0/1281682018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 msgr2=0x55ab1cdcdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842b400640 1 -- 172.21.15.98:0/1281682018 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab1ccfc1f0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.007+0000 7f842be00640 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.008+0000 7f842b400640 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a550f214b5492e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.008+0000 7f8420c00640 1 -- 172.21.15.98:0/1281682018 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c02cc40 con 0x55ab1cdce2b0 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.008+0000 7f8420c00640 1 -- 172.21.15.98:0/1281682018 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f841c005040 con 0x55ab1cdce2b0 2024-03-16T12:23:21.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.008+0000 7f8420c00640 1 -- 172.21.15.98:0/1281682018 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c004920 con 0x55ab1cdce2b0 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f8423e00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 0x7f84100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- 172.21.15.98:0/1281682018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 msgr2=0x55ab1cdd2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c0095e0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f8423e00640 1 -- 172.21.15.98:0/493111238 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84180cf2f0 con 0x7f842c104bf0 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- 172.21.15.98:0/1281682018 shutdown_connections 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f842be00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 0x7f84100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdd3420 0x55ab1cdd58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd2db0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 --2- 172.21.15.98:0/1281682018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdcdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- 172.21.15.98:0/1281682018 >> 172.21.15.98:0/1281682018 conn(0x55ab1cdc8e30 msgr2=0x55ab1cdc9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- 172.21.15.98:0/1281682018 shutdown_connections 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- 172.21.15.98:0/1281682018 wait complete. 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.009+0000 7f84353e5740 1 -- start start 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f842be00640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 0x7f84100a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55ab1cdceb10 tx=0x7f8424009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 Processor -- start 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 0x55ab1cddf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab1ccee110 con 0x55ab1cdd3420 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ab1cd2e4f0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab1cdd6370 con 0x55ab1cdcd8c0 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f84353e5740 10 client.0 _ll_register_callbacks cb 0x55ab1ccdbb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f842b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f842aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f842b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49922/0 (socket says 172.21.15.98:49922) 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.010+0000 7f842b400640 1 -- 172.21.15.98:0/1302270937 learned_addr learned my addr 172.21.15.98:0/1302270937 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842be00640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 0x55ab1cddf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 -- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 msgr2=0x55ab1cdd7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 -- 172.21.15.98:0/1302270937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 msgr2=0x55ab1cddf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 0x55ab1cddf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 -- 172.21.15.98:0/1302270937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab1ca463f0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842b400640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842be00640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 0x55ab1cddf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f842aa00640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd8110 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f841809b630 tx=0x7f841809b530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de21d63106667181 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f8415600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180a3e40 con 0x55ab1cdce2b0 2024-03-16T12:23:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f841c005040 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f841809ad80 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f84180862d0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f84353e5740 10 client.25838 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.011+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ab1cb107d0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f841c005040 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f841809a7f0 con 0x55ab1cdce2b0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 10 client.25838 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 10 client.25838 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 20 client.25838 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cde39a0 0x55ab1cde5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 4 client.25838 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab1cd3f990 con 0x55ab1cde39a0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.012+0000 7f842b400640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cde39a0 0x55ab1cde5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.013+0000 7f842b400640 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cde39a0 0x55ab1cde5db0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52d849b4203dc59b server_cookie=e732def85cd11af9 in_seq=0 out_seq=0 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.013+0000 7f8415600640 10 client.25838 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.013+0000 7f8415600640 1 -- 172.21.15.98:0/1302270937 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab1cd3f990 con 0x55ab1cde39a0 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.013+0000 7f8415600640 10 client.25838 handle_command_reply: tid=0 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 client.25838 shutdown 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 msgr2=0x55ab1cdd8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd8110 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f841809b630 tx=0x7f841809b530 comp rx=0 tx=0).stop 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 shutdown_connections 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cde39a0 0x55ab1cde5db0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdd3420 0x55ab1cddf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f8415600640 0 client.25838 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdd8110 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 --2- 172.21.15.98:0/1302270937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cdd7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 >> 172.21.15.98:0/1302270937 conn(0x55ab1cdc8e30 msgr2=0x55ab1cdc81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 shutdown_connections 2024-03-16T12:23:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 1 -- 172.21.15.98:0/1302270937 wait complete. 2024-03-16T12:23:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.014+0000 7f84353e5740 20 client.25838 trim_cache size 0 max 16384 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 Processor -- start 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 -- start start 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 0x55ab1cdc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1aecd0 con 0x55ab1cdc8e30 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1ac5a0 con 0x55ab1cdce2b0 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f84353e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1aa3f0 con 0x55ab1cdcd8c0 2024-03-16T12:23:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f842aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.029+0000 7f842be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 0x55ab1cdc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44430/0 (socket says 172.21.15.98:44430) 2024-03-16T12:23:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 -- 172.21.15.98:0/1295003475 learned_addr learned my addr 172.21.15.98:0/1295003475 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49930/0 (socket says 172.21.15.98:49930) 2024-03-16T12:23:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 -- 172.21.15.98:0/1295003475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 msgr2=0x55ab1cddf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 -- 172.21.15.98:0/1295003475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 msgr2=0x55ab1cdc8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 0x55ab1cdc8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 -- 172.21.15.98:0/1295003475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab1cdc4c10 con 0x55ab1cdce2b0 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842aa00640 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842b400640 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 0x55ab1cdc8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f842be00640 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8424003c10 tx=0x7f8424016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982bc530e2917acc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.030+0000 7f8417e00640 1 -- 172.21.15.98:0/1295003475 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84240047b0 con 0x55ab1cdce2b0 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.031+0000 7f8417e00640 1 -- 172.21.15.98:0/1295003475 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8424026030 con 0x55ab1cdce2b0 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.031+0000 7f8417e00640 1 -- 172.21.15.98:0/1295003475 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424022800 con 0x55ab1cdce2b0 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.031+0000 7f84353e5740 1 -- 172.21.15.98:0/1295003475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 msgr2=0x55ab1cdea6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.031+0000 7f84353e5740 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8424003c10 tx=0x7f8424016000 comp rx=0 tx=0).stop 2024-03-16T12:23:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 -- 172.21.15.98:0/1295003475 shutdown_connections 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdce2b0 0x55ab1cdea6b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 --2- 172.21.15.98:0/1295003475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdc8e30 0x55ab1cdc8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 -- 172.21.15.98:0/1295003475 >> 172.21.15.98:0/1295003475 conn(0x55ab1cdcf090 msgr2=0x55ab1cdc7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 -- 172.21.15.98:0/1295003475 shutdown_connections 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 -- 172.21.15.98:0/1295003475 wait complete. 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 -- start start 2024-03-16T12:23:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.032+0000 7f84353e5740 1 Processor -- start 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdce2b0 0x55ab1cddde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1aaa60 con 0x55ab1cdce2b0 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1abed0 con 0x55ab1cdc8e30 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f842c1ab130 con 0x55ab1cdcd8c0 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f84353e5740 10 client.0 _ll_register_callbacks cb 0x7f841800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44444/0 (socket says 172.21.15.98:44444) 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842b400640 1 -- 172.21.15.98:0/3135008327 learned_addr learned my addr 172.21.15.98:0/3135008327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49932/0 (socket says 172.21.15.98:49932) 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.033+0000 7f842be00640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdce2b0 0x55ab1cddde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 -- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 msgr2=0x55ab1cddce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 -- 172.21.15.98:0/3135008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdce2b0 msgr2=0x55ab1cddde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdce2b0 0x55ab1cddde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 -- 172.21.15.98:0/3135008327 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab1cdde410 con 0x55ab1cdc8e30 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842aa00640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f842b400640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f841c02c170 tx=0x7f841c005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782c6b57f7c6d2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f8415600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c005310 con 0x55ab1cdc8e30 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8424026030 con 0x55ab1cdc8e30 2024-03-16T12:23:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f841c004050 con 0x55ab1cdc8e30 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f841c005630 con 0x55ab1cdc8e30 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f84353e5740 10 client.25844 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.034+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ab1ce49e20 con 0x55ab1cdc8e30 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8424026030 con 0x55ab1cdc8e30 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f841c0038c0 con 0x55ab1cdc8e30 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 10 client.25844 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 10 client.25844 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 20 client.25844 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cdd3420 0x55ab1ce4c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 4 client.25844 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab1cd39ce0 con 0x55ab1cdd3420 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f842aa00640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cdd3420 0x55ab1ce4c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f842aa00640 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cdd3420 0x55ab1ce4c400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e955bc49d225cb39 server_cookie=dd9c10f6ebd0cd7 in_seq=0 out_seq=0 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.035+0000 7f8415600640 10 client.25844 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f8415600640 1 -- 172.21.15.98:0/3135008327 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab1cd39ce0 con 0x55ab1cdd3420 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f8415600640 10 client.25844 handle_command_reply: tid=0 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 client.25844 shutdown 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 msgr2=0x55ab1cddc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f841c02c170 tx=0x7f841c005e50 comp rx=0 tx=0).stop 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 shutdown_connections 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55ab1cdd3420 0x55ab1ce4c400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab1cdce2b0 0x55ab1cddde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab1cdcd8c0 0x55ab1cddce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 --2- 172.21.15.98:0/3135008327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ab1cdc8e30 0x55ab1cddc920 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f8415600640 0 client.25844 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 >> 172.21.15.98:0/3135008327 conn(0x55ab1cdcf090 msgr2=0x55ab1cdcc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 shutdown_connections 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 1 -- 172.21.15.98:0/3135008327 wait complete. 2024-03-16T12:23:22.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.036+0000 7f84353e5740 20 client.25844 trim_cache size 0 max 16384 2024-03-16T12:23:22.021 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 msgr2=0x7f84100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 0x7f84100a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55ab1cdceb10 tx=0x7f8424009600 comp rx=0 tx=0).stop 2024-03-16T12:23:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 msgr2=0x7f842c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f841800e9f0 tx=0x7f841800edf0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 shutdown_connections 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f84100a2800 0x7f84100a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f842c1055b0 0x7f842c1a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f842c104bf0 0x7f842c19a280 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 --2- 172.21.15.98:0/493111238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f842c103a00 0x7f842c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 >> 172.21.15.98:0/493111238 conn(0x7f842c0fef90 msgr2=0x7f842c107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 shutdown_connections 2024-03-16T12:23:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.037+0000 7f8432600640 1 -- 172.21.15.98:0/493111238 wait complete. 2024-03-16T12:23:22.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T12:23:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2255066134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48075820 msgr2=0x7f1e48073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2255066134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48075820 0x7f1e48073c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009900 tx=0x7f1e3802bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2255066134 shutdown_connections 2024-03-16T12:23:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2255066134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e481094a0 0x7f1e4810b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2255066134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e48074240 0x7f1e480746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2255066134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48075820 0x7f1e48073c70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2255066134 >> 172.21.15.98:0/2255066134 conn(0x7f1e480fcb70 msgr2=0x7f1e480fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.253+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2255066134 shutdown_connections 2024-03-16T12:23:22.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.254+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2255066134 wait complete. 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.254+0000 7f1e4ce00640 1 Processor -- start 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 -- start start 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 0x7f1e48199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481094a0 0x7f1e481a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810f470 con 0x7f1e481094a0 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810f2f0 con 0x7f1e48074240 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e4ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4810f5f0 con 0x7f1e48075820 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e47400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 0x7f1e48199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481094a0 0x7f1e481a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49950/0 (socket says 172.21.15.98:49950) 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e47400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 0x7f1e48199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44448/0 (socket says 172.21.15.98:44448) 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.255+0000 7f1e46a00640 1 -- 172.21.15.98:0/2520508996 learned_addr learned my addr 172.21.15.98:0/2520508996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 -- 172.21.15.98:0/2520508996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 msgr2=0x7f1e48199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 0x7f1e48199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 -- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481094a0 msgr2=0x7f1e481a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481094a0 0x7f1e481a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 -- 172.21.15.98:0/2520508996 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e481a16e0 con 0x7f1e48075820 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e46a00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400e9f0 tx=0x7f1e3400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e3fe00640 1 -- 172.21.15.98:0/2520508996 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400ccc0 con 0x7f1e48075820 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e3fe00640 1 -- 172.21.15.98:0/2520508996 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e34010910 con 0x7f1e48075820 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e481a19d0 con 0x7f1e48075820 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e3fe00640 1 -- 172.21.15.98:0/2520508996 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e340188a0 con 0x7f1e48075820 2024-03-16T12:23:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.256+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e481a1f20 con 0x7f1e48075820 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.259+0000 7f1e5116c740 1 Processor -- start 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 -- start start 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e3fe00640 1 -- 172.21.15.98:0/2520508996 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1e34010ac0 con 0x7f1e48075820 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56863f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56866a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555c568263b0 con 0x555c5685f480 2024-03-16T12:23:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c567c3450 con 0x555c568645f0 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e5116c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555c5676bcb0 con 0x555c5685ea90 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49962/0 (socket says 172.21.15.98:49962) 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e47e00640 1 -- 172.21.15.98:0/1204204857 learned_addr learned my addr 172.21.15.98:0/1204204857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e46a00640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56866a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e47400640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56863f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.260+0000 7f1e47e00640 1 -- 172.21.15.98:0/1204204857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 msgr2=0x555c56866a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47e00640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56866a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47e00640 1 -- 172.21.15.98:0/1204204857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 msgr2=0x555c56863f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47e00640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56863f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47e00640 1 -- 172.21.15.98:0/1204204857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c5678d1f0 con 0x555c5685ea90 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e46a00640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56866a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47400640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56863f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e47e00640 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1e40011440 tx=0x7f1e40011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2237611a3add979 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e3cc00640 1 -- 172.21.15.98:0/1204204857 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40003c30 con 0x555c5685ea90 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e3cc00640 1 -- 172.21.15.98:0/1204204857 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e40003dd0 con 0x555c5685ea90 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e3cc00640 1 -- 172.21.15.98:0/1204204857 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4000faf0 con 0x555c5685ea90 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.261+0000 7f1e3fe00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e3fe00640 1 -- 172.21.15.98:0/2520508996 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e340cf260 con 0x7f1e48075820 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e47400640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 -- 172.21.15.98:0/1204204857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 msgr2=0x555c5685eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1e40011440 tx=0x7f1e40011900 comp rx=0 tx=0).stop 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 -- 172.21.15.98:0/1204204857 shutdown_connections 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56866a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56863f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1204204857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5685eeb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 -- 172.21.15.98:0/1204204857 >> 172.21.15.98:0/1204204857 conn(0x555c5685a000 msgr2=0x555c5685a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e47400640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e3802c240 tx=0x7f1e3802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 -- 172.21.15.98:0/1204204857 shutdown_connections 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.262+0000 7f1e5116c740 1 -- 172.21.15.98:0/1204204857 wait complete. 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.263+0000 7f1e5116c740 1 -- start start 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.263+0000 7f1e5116c740 1 Processor -- start 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56868e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56869340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56870310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555c5677f110 con 0x555c5685f480 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c567bf4f0 con 0x555c568645f0 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555c568675a0 con 0x555c5685ea90 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e5116c740 10 client.0 _ll_register_callbacks cb 0x555c5676cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56868e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56868e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49966/0 (socket says 172.21.15.98:49966) 2024-03-16T12:23:22.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56870310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47400640 1 -- 172.21.15.98:0/395743236 learned_addr learned my addr 172.21.15.98:0/395743236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e46a00640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56869340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47e00640 1 -- 172.21.15.98:0/395743236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 msgr2=0x555c56868e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47e00640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56868e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47e00640 1 -- 172.21.15.98:0/395743236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 msgr2=0x555c56869340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.264+0000 7f1e47e00640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56869340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e47e00640 1 -- 172.21.15.98:0/395743236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c564d73f0 con 0x555c568645f0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e46a00640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56869340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e47e00640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56870310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1e4001b040 tx=0x7f1e40004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=517fff830ec13f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e31600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40029020 con 0x555c568645f0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e4000fc80 con 0x555c568645f0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e40003dd0 con 0x555c568645f0 2024-03-16T12:23:22.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1e40034050 con 0x555c568645f0 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e5116c740 10 client.25856 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.265+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c565a17d0 con 0x555c568645f0 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e40003dd0 con 0x555c568645f0 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1e40005160 con 0x555c568645f0 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 10 client.25856 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 10 client.25856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 20 client.25856 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c56874a90 0x555c56876ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 4 client.25856 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c567d0990 con 0x555c56874a90 2024-03-16T12:23:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.266+0000 7f1e47400640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c56874a90 0x555c56876ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.267+0000 7f1e47400640 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c56874a90 0x555c56876ea0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7075eda4dae55a8 server_cookie=bc01093ee00ab73f in_seq=0 out_seq=0 2024-03-16T12:23:22.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.267+0000 7f1e31600640 10 client.25856 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.267+0000 7f1e31600640 1 -- 172.21.15.98:0/395743236 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555c567d0990 con 0x555c56874a90 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.267+0000 7f1e31600640 10 client.25856 handle_command_reply: tid=0 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.267+0000 7f1e5116c740 1 client.25856 shutdown 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 msgr2=0x555c56870310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56870310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1e4001b040 tx=0x7f1e40004c70 comp rx=0 tx=0).stop 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 shutdown_connections 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c568645f0 0x555c56870310 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c5685f480 0x555c56869340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c56874a90 0x555c56876ea0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 --2- 172.21.15.98:0/395743236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56868e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 >> 172.21.15.98:0/395743236 conn(0x555c5685a000 msgr2=0x555c56859220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e31600640 0 client.25856 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 shutdown_connections 2024-03-16T12:23:22.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 1 -- 172.21.15.98:0/395743236 wait complete. 2024-03-16T12:23:22.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.268+0000 7f1e5116c740 20 client.25856 trim_cache size 0 max 16384 2024-03-16T12:23:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.282+0000 7f1e5116c740 1 Processor -- start 2024-03-16T12:23:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 -- start start 2024-03-16T12:23:22.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c56870230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c56858990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481aecf0 con 0x555c568645f0 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481ac5c0 con 0x555c5685f480 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e5116c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481aa410 con 0x555c5685ea90 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c56870230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c56858990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49968/0 (socket says 172.21.15.98:49968) 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47400640 1 -- 172.21.15.98:0/1216030788 learned_addr learned my addr 172.21.15.98:0/1216030788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47400640 1 -- 172.21.15.98:0/1216030788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 msgr2=0x555c56870230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.283+0000 7f1e47400640 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c56870230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e47400640 1 -- 172.21.15.98:0/1216030788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 msgr2=0x555c56858990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e47400640 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c56858990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e47400640 1 -- 172.21.15.98:0/1216030788 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c56855850 con 0x555c5685ea90 2024-03-16T12:23:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e47e00640 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c56858990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e47400640 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1e38036040 tx=0x7f1e38041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d5a497c8fb65006 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e33e00640 1 -- 172.21.15.98:0/1216030788 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38065070 con 0x555c5685ea90 2024-03-16T12:23:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e33e00640 1 -- 172.21.15.98:0/1216030788 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e38060d30 con 0x555c5685ea90 2024-03-16T12:23:22.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.284+0000 7f1e33e00640 1 -- 172.21.15.98:0/1216030788 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38045070 con 0x555c5685ea90 2024-03-16T12:23:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- 172.21.15.98:0/1216030788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 msgr2=0x555c56855080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1e38036040 tx=0x7f1e38041000 comp rx=0 tx=0).stop 2024-03-16T12:23:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- 172.21.15.98:0/1216030788 shutdown_connections 2024-03-16T12:23:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c56858990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c56870230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 --2- 172.21.15.98:0/1216030788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c56855080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- 172.21.15.98:0/1216030788 >> 172.21.15.98:0/1216030788 conn(0x555c5685a000 msgr2=0x555c56868e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- 172.21.15.98:0/1216030788 shutdown_connections 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- 172.21.15.98:0/1216030788 wait complete. 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.285+0000 7f1e5116c740 1 -- start start 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 Processor -- start 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c5686d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c5686e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481aaa80 con 0x555c568645f0 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481abef0 con 0x555c5685f480 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481ab150 con 0x555c5685ea90 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e5116c740 10 client.0 _ll_register_callbacks cb 0x7f1e40009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:22.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c5686d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49974/0 (socket says 172.21.15.98:49974) 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 -- 172.21.15.98:0/3505487700 learned_addr learned my addr 172.21.15.98:0/3505487700 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47e00640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c5686e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 -- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 msgr2=0x555c5686d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c5686d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 -- 172.21.15.98:0/3505487700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 msgr2=0x555c5686e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c5686e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.286+0000 7f1e47400640 1 -- 172.21.15.98:0/3505487700 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c5686ef40 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e46a00640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c5686d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e47400640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1e3802bc70 tx=0x7f1e38004380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e913bde2f9e3cdd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e31600640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38065070 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e380047c0 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e38060d30 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1e38073050 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e5116c740 10 client.25852 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.287+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c56870240 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e38060d30 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1e38004eb0 con 0x555c5685ea90 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 10 client.25852 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 10 client.25852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:22.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 20 client.25852 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c568daff0 0x555c568dd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 4 client.25852 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c567cace0 con 0x555c568daff0 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e46a00640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c568daff0 0x555c568dd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e46a00640 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c568daff0 0x555c568dd3e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f076dce15567067a server_cookie=b1479b1fe316fa49 in_seq=0 out_seq=0 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.288+0000 7f1e31600640 10 client.25852 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e31600640 1 -- 172.21.15.98:0/3505487700 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c567cace0 con 0x555c568daff0 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e31600640 10 client.25852 handle_command_reply: tid=0 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 client.25852 shutdown 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 msgr2=0x555c5686d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1e3802bc70 tx=0x7f1e38004380 comp rx=0 tx=0).stop 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 shutdown_connections 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c568645f0 0x555c5686e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c5685f480 0x555c5686d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x555c568daff0 0x555c568dd3e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 --2- 172.21.15.98:0/3505487700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555c5685ea90 0x555c5686d400 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e31600640 0 client.25852 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 >> 172.21.15.98:0/3505487700 conn(0x555c5685a000 msgr2=0x555c5685d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 shutdown_connections 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 1 -- 172.21.15.98:0/3505487700 wait complete. 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.289+0000 7f1e5116c740 20 client.25852 trim_cache size 0 max 16384 2024-03-16T12:23:22.274 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 msgr2=0x7f1e2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e3802c240 tx=0x7f1e3802df90 comp rx=0 tx=0).stop 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 msgr2=0x7f1e4819a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400e9f0 tx=0x7f1e3400edf0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 shutdown_connections 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e481094a0 0x7f1e481a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e48075820 0x7f1e4819a000 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 --2- 172.21.15.98:0/2520508996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e48074240 0x7f1e48199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 >> 172.21.15.98:0/2520508996 conn(0x7f1e480fcb70 msgr2=0x7f1e481053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 shutdown_connections 2024-03-16T12:23:22.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:22.290+0000 7f1e4ce00640 1 -- 172.21.15.98:0/2520508996 wait complete. 2024-03-16T12:23:22.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:22 smithi098 ceph-mon[31633]: pgmap v460: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 775 KiB/s wr, 8 op/s 2024-03-16T12:23:22.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:22 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1401960386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:22.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:22 smithi173 ceph-mon[36632]: pgmap v460: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 775 KiB/s wr, 8 op/s 2024-03-16T12:23:22.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:22 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1401960386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:22.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:22 smithi153 ceph-mon[36901]: pgmap v460: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 775 KiB/s wr, 8 op/s 2024-03-16T12:23:22.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:22 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1401960386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:23.287 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T12:23:23.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:23:23.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 -- 172.21.15.98:0/3017298147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 msgr2=0x7f19381004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 --2- 172.21.15.98:0/3017298147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f19381004b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1928009860 tx=0x7f192802bbc0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 -- 172.21.15.98:0/3017298147 shutdown_connections 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 --2- 172.21.15.98:0/3017298147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1938101390 0x7f1938101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 --2- 172.21.15.98:0/3017298147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19381009f0 0x7f1938100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 --2- 172.21.15.98:0/3017298147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f19381004b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 -- 172.21.15.98:0/3017298147 >> 172.21.15.98:0/3017298147 conn(0x7f19380fbc70 msgr2=0x7f19380fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 -- 172.21.15.98:0/3017298147 shutdown_connections 2024-03-16T12:23:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.522+0000 7f193ca00640 1 -- 172.21.15.98:0/3017298147 wait complete. 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.523+0000 7f193ca00640 1 Processor -- start 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.523+0000 7f193ca00640 1 -- start start 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.523+0000 7f193ca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f1938199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.523+0000 7f193ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 0x7f193819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.523+0000 7f193ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f193ca00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f193810f680 con 0x7f1938100090 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f193ca00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f193810f500 con 0x7f19381009f0 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f193ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f193810f800 con 0x7f1938101390 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1935e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f1938199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f192fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 0x7f193819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50000/0 (socket says 172.21.15.98:50000) 2024-03-16T12:23:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 -- 172.21.15.98:0/2027089750 learned_addr learned my addr 172.21.15.98:0/2027089750 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1935e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f1938199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56248/0 (socket says 172.21.15.98:56248) 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 -- 172.21.15.98:0/2027089750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 msgr2=0x7f193819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 0x7f193819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 -- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 msgr2=0x7f1938199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f1938199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 -- 172.21.15.98:0/2027089750 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19381a1960 con 0x7f1938101390 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f192fe00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 0x7f193819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.524+0000 7f1936800640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1930009fd0 tx=0x7f193000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.525+0000 7f192ea00640 1 -- 172.21.15.98:0/2027089750 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930004b20 con 0x7f1938101390 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.525+0000 7f192ea00640 1 -- 172.21.15.98:0/2027089750 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1930003e50 con 0x7f1938101390 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.525+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19381a1c50 con 0x7f1938101390 2024-03-16T12:23:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.525+0000 7f192ea00640 1 -- 172.21.15.98:0/2027089750 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000b630 con 0x7f1938101390 2024-03-16T12:23:23.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.525+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381a2150 con 0x7f1938101390 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f192ea00640 1 -- 172.21.15.98:0/2027089750 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f193001d020 con 0x7f1938101390 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 Processor -- start 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 -- start start 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a879cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a87c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56249a7d7f80 con 0x56249a8751d0 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56249a780ad0 con 0x56249a87a340 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f193f7b7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56249a7d4020 con 0x56249a8747e0 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.528+0000 7f192fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50004/0 (socket says 172.21.15.98:50004) 2024-03-16T12:23:23.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 -- 172.21.15.98:0/3543973819 learned_addr learned my addr 172.21.15.98:0/3543973819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f1935e00640 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a879cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a87c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 -- 172.21.15.98:0/3543973819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 msgr2=0x56249a87c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a87c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 -- 172.21.15.98:0/3543973819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 msgr2=0x56249a879cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a879cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 -- 172.21.15.98:0/3543973819 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249a7986f0 con 0x56249a8747e0 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192fe00640 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1920013c80 tx=0x7f1920009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4bcd45d5f44021c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192ea00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 0x7f191c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f1927000640 1 -- 172.21.15.98:0/3543973819 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920013d70 con 0x56249a8747e0 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f192ea00640 1 -- 172.21.15.98:0/2027089750 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19300cee40 con 0x7f1938101390 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f1927000640 1 -- 172.21.15.98:0/3543973819 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1920013170 con 0x56249a8747e0 2024-03-16T12:23:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.529+0000 7f1927000640 1 -- 172.21.15.98:0/3543973819 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920010d60 con 0x56249a8747e0 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.530+0000 7f1935e00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 0x7f191c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.530+0000 7f193f7b7740 1 -- 172.21.15.98:0/3543973819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 msgr2=0x56249a874c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.530+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1920013c80 tx=0x7f1920009c00 comp rx=0 tx=0).stop 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.530+0000 7f1935e00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 0x7f191c0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f193819b220 tx=0x7f192802cc40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 -- 172.21.15.98:0/3543973819 shutdown_connections 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a87c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a879cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3543973819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a874c00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 -- 172.21.15.98:0/3543973819 >> 172.21.15.98:0/3543973819 conn(0x56249a83c930 msgr2=0x56249a870150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 -- 172.21.15.98:0/3543973819 shutdown_connections 2024-03-16T12:23:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 -- 172.21.15.98:0/3543973819 wait complete. 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 -- start start 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.531+0000 7f193f7b7740 1 Processor -- start 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 0x56249a87eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 0x56249a87f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a8860f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56249a87d2f0 con 0x56249a8747e0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56249a793f30 con 0x56249a87a340 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56249a87d470 con 0x56249a8751d0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f193f7b7740 10 client.0 _ll_register_callbacks cb 0x56249a781bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f192fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 0x56249a87f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1935e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 0x56249a87eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a8860f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1935e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 0x56249a87eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56264/0 (socket says 172.21.15.98:56264) 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1935e00640 1 -- 172.21.15.98:0/192073963 learned_addr learned my addr 172.21.15.98:0/192073963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 -- 172.21.15.98:0/192073963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 msgr2=0x56249a87f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 0x56249a87f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 -- 172.21.15.98:0/192073963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 msgr2=0x56249a87eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 0x56249a87eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f1936800640 1 -- 172.21.15.98:0/192073963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249a5b47e0 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.532+0000 7f192fe00640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 0x56249a87f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f1936800640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a8860f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f193009b120 tx=0x7f19300976f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2cd8cecbe8a760c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193001a070 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1920013170 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f193009f160 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1930099560 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 10 client.25874 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56249a614f70 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1920013170 con 0x56249a87a340 2024-03-16T12:23:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f193000b040 con 0x56249a87a340 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 10 client.25874 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 10 client.25874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 20 client.25874 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a88a900 0x56249a88cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 4 client.25874 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.533+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56249a7e54c0 con 0x56249a88a900 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.534+0000 7f1935e00640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a88a900 0x56249a88cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.534+0000 7f1935e00640 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a88a900 0x56249a88cd60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7090b2571517cddb server_cookie=8e9ec99ffb158616 in_seq=0 out_seq=0 2024-03-16T12:23:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.534+0000 7f191b400640 10 client.25874 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.535+0000 7f191b400640 1 -- 172.21.15.98:0/192073963 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56249a7e54c0 con 0x56249a88a900 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.535+0000 7f191b400640 10 client.25874 handle_command_reply: tid=0 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.535+0000 7f193f7b7740 1 client.25874 shutdown 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 msgr2=0x56249a8860f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a8860f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f193009b120 tx=0x7f19300976f0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 shutdown_connections 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a8860f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a88a900 0x56249a88cd60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8751d0 0x56249a87f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f191b400640 0 client.25874 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 --2- 172.21.15.98:0/192073963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8747e0 0x56249a87eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 >> 172.21.15.98:0/192073963 conn(0x56249a83c930 msgr2=0x56249a86f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 shutdown_connections 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 1 -- 172.21.15.98:0/192073963 wait complete. 2024-03-16T12:23:23.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.536+0000 7f193f7b7740 20 client.25874 trim_cache size 0 max 16384 2024-03-16T12:23:23.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 Processor -- start 2024-03-16T12:23:23.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 -- start start 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a86eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a8860b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19381aedd0 con 0x56249a8751d0 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19381ac6a0 con 0x56249a87a340 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.555+0000 7f193f7b7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19381aa4f0 con 0x56249a8747e0 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44544/0 (socket says 172.21.15.98:44544) 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 -- 172.21.15.98:0/2403930528 learned_addr learned my addr 172.21.15.98:0/2403930528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1935e00640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a86eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f192fe00640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a8860b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 -- 172.21.15.98:0/2403930528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 msgr2=0x56249a86eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a86eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 -- 172.21.15.98:0/2403930528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 msgr2=0x56249a8860b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a8860b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 -- 172.21.15.98:0/2403930528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249a4ac170 con 0x56249a87a340 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f192fe00640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a8860b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.556+0000 7f1936800640 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1930009c00 tx=0x7f193000ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79e3979c3260ae03 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f1926600640 1 -- 172.21.15.98:0/2403930528 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193001a070 con 0x56249a87a340 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f1926600640 1 -- 172.21.15.98:0/2403930528 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300bb3d0 con 0x56249a87a340 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f1926600640 1 -- 172.21.15.98:0/2403930528 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193009f570 con 0x56249a87a340 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 -- 172.21.15.98:0/2403930528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 msgr2=0x56249a86f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1930009c00 tx=0x7f193000ceb0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 -- 172.21.15.98:0/2403930528 shutdown_connections 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a87a340 0x56249a86f000 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a8860b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 --2- 172.21.15.98:0/2403930528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a8747e0 0x56249a86eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.557+0000 7f193f7b7740 1 -- 172.21.15.98:0/2403930528 >> 172.21.15.98:0/2403930528 conn(0x56249a83c930 msgr2=0x56249a876530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 -- 172.21.15.98:0/2403930528 shutdown_connections 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 -- 172.21.15.98:0/2403930528 wait complete. 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 -- start start 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 Processor -- start 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 0x56249a883690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a883bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.558+0000 7f193f7b7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f193f7b7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19381aab60 con 0x56249a8751d0 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f193f7b7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19381abfd0 con 0x56249a8747e0 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f193f7b7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19381ab230 con 0x56249a87a340 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f193f7b7740 10 client.0 _ll_register_callbacks cb 0x7f1930009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1935e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 0x56249a883690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50038/0 (socket says 172.21.15.98:50038) 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 -- 172.21.15.98:0/3409910097 learned_addr learned my addr 172.21.15.98:0/3409910097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f192fe00640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a883bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 -- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 msgr2=0x56249a883690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 0x56249a883690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 -- 172.21.15.98:0/3409910097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 msgr2=0x56249a883bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a883bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 -- 172.21.15.98:0/3409910097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249a885130 con 0x56249a87a340 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1935e00640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 0x56249a883690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f1936800640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f19300977f0 tx=0x7f1930004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55562a61429982d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.559+0000 7f191b400640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193001a070 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f193009f160 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19300bb3d0 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f193009f360 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 10 client.25870 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56249a8863e0 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19300bb3d0 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f193009a870 con 0x56249a87a340 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 10 client.25870 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 10 client.25870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 20 client.25870 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:23.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a8f1070 0x56249a8f3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 4 client.25870 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.560+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56249a7df810 con 0x56249a8f1070 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.561+0000 7f1935e00640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a8f1070 0x56249a8f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.561+0000 7f1935e00640 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a8f1070 0x56249a8f3490 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a02b4c75f4a6ebf2 server_cookie=73b7046b1a2bc6de in_seq=0 out_seq=0 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.561+0000 7f191b400640 10 client.25870 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f191b400640 1 -- 172.21.15.98:0/3409910097 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56249a7df810 con 0x56249a8f1070 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f191b400640 10 client.25870 handle_command_reply: tid=0 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 client.25870 shutdown 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 msgr2=0x56249a884b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f19300977f0 tx=0x7f1930004310 comp rx=0 tx=0).stop 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 shutdown_connections 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56249a8f1070 0x56249a8f3490 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56249a87a340 0x56249a884b40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56249a8751d0 0x56249a883bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f191b400640 0 client.25870 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 --2- 172.21.15.98:0/3409910097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249a8747e0 0x56249a883690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 >> 172.21.15.98:0/3409910097 conn(0x56249a83c930 msgr2=0x56249a873390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.562+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 shutdown_connections 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193f7b7740 1 -- 172.21.15.98:0/3409910097 wait complete. 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193f7b7740 20 client.25870 trim_cache size 0 max 16384 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:23.549 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-03-16T12:23:23.550 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "49488d6a-af6b-4125-8a8c-42313ce21671", 2024-03-16T12:23:23.550 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-03-16T12:23:23.550 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 msgr2=0x7f191c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 0x7f191c0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f193819b220 tx=0x7f192802cc40 comp rx=0 tx=0).stop 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 msgr2=0x7f19381a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1930009fd0 tx=0x7f193000c800 comp rx=0 tx=0).stop 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 shutdown_connections 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f191c0a2800 0x7f191c0a4cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1938101390 0x7f19381a12c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19381009f0 0x7f193819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 --2- 172.21.15.98:0/2027089750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1938100090 0x7f1938199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.563+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 >> 172.21.15.98:0/2027089750 conn(0x7f19380fbc70 msgr2=0x7f19380fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.564+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 shutdown_connections 2024-03-16T12:23:23.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.564+0000 7f193ca00640 1 -- 172.21.15.98:0/2027089750 wait complete. 2024-03-16T12:23:23.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:23:23.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:23:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.784+0000 7f548d200640 1 -- 172.21.15.98:0/2902183725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881017f0 msgr2=0x7f5488101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.784+0000 7f548d200640 1 --2- 172.21.15.98:0/2902183725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881017f0 0x7f5488101c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5478009900 tx=0x7f547802bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 -- 172.21.15.98:0/2902183725 shutdown_connections 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 --2- 172.21.15.98:0/2902183725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881033a0 0x7f5488109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 --2- 172.21.15.98:0/2902183725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881029e0 0x7f5488102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 --2- 172.21.15.98:0/2902183725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881017f0 0x7f5488101c10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 -- 172.21.15.98:0/2902183725 >> 172.21.15.98:0/2902183725 conn(0x7f54880fcd80 msgr2=0x7f54880ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 -- 172.21.15.98:0/2902183725 shutdown_connections 2024-03-16T12:23:23.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.785+0000 7f548d200640 1 -- 172.21.15.98:0/2902183725 wait complete. 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 Processor -- start 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 -- start start 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881029e0 0x7f548819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 0x7f54881a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f548810d4c0 con 0x7f54881033a0 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f548810d340 con 0x7f54881017f0 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.786+0000 7f548d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f548810d640 con 0x7f54881029e0 2024-03-16T12:23:23.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44576/0 (socket says 172.21.15.98:44576) 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 -- 172.21.15.98:0/137814288 learned_addr learned my addr 172.21.15.98:0/137814288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5486800640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 0x7f54881a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485400640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881029e0 0x7f548819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 -- 172.21.15.98:0/137814288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881029e0 msgr2=0x7f548819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881029e0 0x7f548819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 -- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 msgr2=0x7f54881a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 0x7f54881a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 -- 172.21.15.98:0/137814288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54881a1990 con 0x7f54881017f0 2024-03-16T12:23:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5486800640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 0x7f54881a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f5485e00640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f547802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.787+0000 7f547ea00640 1 -- 172.21.15.98:0/137814288 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478043020 con 0x7f54881017f0 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.788+0000 7f547ea00640 1 -- 172.21.15.98:0/137814288 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54780054e0 con 0x7f54881017f0 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.788+0000 7f547ea00640 1 -- 172.21.15.98:0/137814288 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478003c40 con 0x7f54881017f0 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.788+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54881a1c20 con 0x7f54881017f0 2024-03-16T12:23:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.788+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54881a2120 con 0x7f54881017f0 2024-03-16T12:23:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f547ea00640 1 -- 172.21.15.98:0/137814288 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5478046020 con 0x7f54881017f0 2024-03-16T12:23:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 Processor -- start 2024-03-16T12:23:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 -- start start 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fe200 0x556418902d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 0x556418905800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5564188621f0 con 0x5564188fe200 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55641880aa80 con 0x5564188fd810 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.791+0000 7f548ff88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55641885e290 con 0x556418903370 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5486800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5486800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44584/0 (socket says 172.21.15.98:44584) 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5486800640 1 -- 172.21.15.98:0/267837145 learned_addr learned my addr 172.21.15.98:0/267837145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 0x556418905800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485e00640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fe200 0x556418902d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 -- 172.21.15.98:0/267837145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 msgr2=0x5564188fdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 -- 172.21.15.98:0/267837145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fe200 msgr2=0x556418902d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fe200 0x556418902d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 -- 172.21.15.98:0/267837145 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564186407d0 con 0x556418903370 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5486800640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5485400640 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 0x556418905800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5474009bf0 tx=0x7f5474011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec50f00aba3e8712 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f547ea00640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 0x7f546c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.792+0000 7f5473400640 1 -- 172.21.15.98:0/267837145 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474004050 con 0x556418903370 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f547ea00640 1 -- 172.21.15.98:0/137814288 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54780f0e70 con 0x7f54881017f0 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f5473400640 1 -- 172.21.15.98:0/267837145 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5474005500 con 0x556418903370 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f5473400640 1 -- 172.21.15.98:0/267837145 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474004e40 con 0x556418903370 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f5486800640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 0x7f546c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f548ff88740 1 -- 172.21.15.98:0/267837145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 msgr2=0x556418905800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f548ff88740 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 0x556418905800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5474009bf0 tx=0x7f5474011810 comp rx=0 tx=0).stop 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.793+0000 7f5486800640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 0x7f546c0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5564188fea60 tx=0x7f5480009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 -- 172.21.15.98:0/267837145 shutdown_connections 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556418903370 0x556418905800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fe200 0x556418902d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 --2- 172.21.15.98:0/267837145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x5564188fdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 -- 172.21.15.98:0/267837145 >> 172.21.15.98:0/267837145 conn(0x5564188f8d80 msgr2=0x5564188f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 -- 172.21.15.98:0/267837145 shutdown_connections 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 -- 172.21.15.98:0/267837145 wait complete. 2024-03-16T12:23:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 -- start start 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.794+0000 7f548ff88740 1 Processor -- start 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x55641890f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556418906320 con 0x556418903370 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55641881ded0 con 0x5564188fd810 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5564189064a0 con 0x5564188fe200 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f548ff88740 10 client.0 _ll_register_callbacks cb 0x556418859220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44590/0 (socket says 172.21.15.98:44590) 2024-03-16T12:23:23.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 -- 172.21.15.98:0/4196796326 learned_addr learned my addr 172.21.15.98:0/4196796326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485400640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50062/0 (socket says 172.21.15.98:50062) 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5486800640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x55641890f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 -- 172.21.15.98:0/4196796326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 msgr2=0x556418908290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 -- 172.21.15.98:0/4196796326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 msgr2=0x55641890f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x55641890f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485e00640 1 -- 172.21.15.98:0/4196796326 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564185763f0 con 0x5564188fd810 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.795+0000 7f5485400640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5485e00640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f54780bd150 tx=0x7f54780b97b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bc4b6de30732324 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478004250 con 0x5564188fd810 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54780ee6c0 con 0x5564188fd810 2024-03-16T12:23:23.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5474005500 con 0x5564188fd810 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f54780035f0 con 0x5564188fd810 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f548ff88740 10 client.25892 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5564185360f0 con 0x5564188fd810 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5474005500 con 0x5564188fd810 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.796+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f54780d6140 con 0x5564188fd810 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 10 client.25892 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 20 client.25892 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x556418913b30 0x556418915fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 4 client.25892 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556418875be0 con 0x556418913b30 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f5485400640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x556418913b30 0x556418915fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f5485400640 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x556418913b30 0x556418915fc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fc788b74aa4d463 server_cookie=1ed2220df02b6350 in_seq=0 out_seq=0 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.797+0000 7f5467e00640 10 client.25892 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f5467e00640 1 -- 172.21.15.98:0/4196796326 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556418875be0 con 0x556418913b30 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f5467e00640 10 client.25892 handle_command_reply: tid=0 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 client.25892 shutdown 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 msgr2=0x556418907d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f54780bd150 tx=0x7f54780b97b0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 shutdown_connections 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x55641890f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x556418913b30 0x556418915fc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x556418908290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 --2- 172.21.15.98:0/4196796326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fd810 0x556418907d50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 >> 172.21.15.98:0/4196796326 conn(0x5564188f8d80 msgr2=0x5564188f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f5467e00640 0 client.25892 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.798+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 shutdown_connections 2024-03-16T12:23:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.799+0000 7f548ff88740 1 -- 172.21.15.98:0/4196796326 wait complete. 2024-03-16T12:23:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.799+0000 7f548ff88740 20 client.25892 trim_cache size 0 max 16384 2024-03-16T12:23:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.817+0000 7f548ff88740 1 Processor -- start 2024-03-16T12:23:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.817+0000 7f548ff88740 1 -- start start 2024-03-16T12:23:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fd810 0x5564188f7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x5564188f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54881aeec0 con 0x556418903370 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54881ac790 con 0x5564188fe200 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f548ff88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f54881aa5e0 con 0x5564188fd810 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:44602/0 (socket says 172.21.15.98:44602) 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 -- 172.21.15.98:0/1728562104 learned_addr learned my addr 172.21.15.98:0/1728562104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485e00640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fd810 0x5564188f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5486800640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x5564188f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 -- 172.21.15.98:0/1728562104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fd810 msgr2=0x5564188f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fd810 0x5564188f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 -- 172.21.15.98:0/1728562104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 msgr2=0x5564188f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x5564188f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5485400640 1 -- 172.21.15.98:0/1728562104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564188f3370 con 0x5564188fe200 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.818+0000 7f5486800640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x5564188f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.819+0000 7f5485400640 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5564188f9cf0 tx=0x7f5474004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a28eddc81c4db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.819+0000 7f5472a00640 1 -- 172.21.15.98:0/1728562104 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547402ecb0 con 0x5564188fe200 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.819+0000 7f5472a00640 1 -- 172.21.15.98:0/1728562104 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5474033070 con 0x5564188fe200 2024-03-16T12:23:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.819+0000 7f5472a00640 1 -- 172.21.15.98:0/1728562104 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5474003b50 con 0x5564188fe200 2024-03-16T12:23:23.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- 172.21.15.98:0/1728562104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 msgr2=0x55641890f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5564188f9cf0 tx=0x7f5474004570 comp rx=0 tx=0).stop 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- 172.21.15.98:0/1728562104 shutdown_connections 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556418903370 0x5564188f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5564188fe200 0x55641890f250 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 --2- 172.21.15.98:0/1728562104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fd810 0x5564188f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- 172.21.15.98:0/1728562104 >> 172.21.15.98:0/1728562104 conn(0x5564188f8d80 msgr2=0x5564188ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- 172.21.15.98:0/1728562104 shutdown_connections 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- 172.21.15.98:0/1728562104 wait complete. 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 -- start start 2024-03-16T12:23:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.820+0000 7f548ff88740 1 Processor -- start 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x55641890d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 0x55641890e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54881aac50 con 0x5564188fd810 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54881ac0c0 con 0x556418903370 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f54881ab320 con 0x5564188fe200 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f548ff88740 10 client.0 _ll_register_callbacks cb 0x7f54780b90b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5486800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 0x55641890e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56326/0 (socket says 172.21.15.98:56326) 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 -- 172.21.15.98:0/2561872197 learned_addr learned my addr 172.21.15.98:0/2561872197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485400640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x55641890d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 -- 172.21.15.98:0/2561872197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 msgr2=0x55641890d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x55641890d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 -- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 msgr2=0x55641890e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 0x55641890e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5485e00640 1 -- 172.21.15.98:0/2561872197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55641890e640 con 0x5564188fd810 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.821+0000 7f5486800640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 0x55641890e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5485e00640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5478006010 tx=0x7f54780bd2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906576f929382ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5467e00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54780bcf90 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54780c5760 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5474033070 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5478004250 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f548ff88740 10 client.15735 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55641897a110 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.822+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5474033070 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f54780bc8a0 con 0x5564188fd810 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 10 client.15735 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 20 client.15735 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55641897a8c0 0x55641897ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 4 client.15735 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55641886f730 con 0x55641897a8c0 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f5485400640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55641897a8c0 0x55641897ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f5467e00640 10 client.15735 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.823+0000 7f5485400640 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55641897a8c0 0x55641897ccb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7af1565f092cf36 server_cookie=2ea7efdf6c4b6843 in_seq=0 out_seq=0 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f5467e00640 1 -- 172.21.15.98:0/2561872197 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55641886f730 con 0x55641897a8c0 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f5467e00640 10 client.15735 handle_command_reply: tid=0 2024-03-16T12:23:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 client.15735 shutdown 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 msgr2=0x55641890cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5478006010 tx=0x7f54780bd2e0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 shutdown_connections 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556418903370 0x55641890e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55641897a8c0 0x55641897ccb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564188fe200 0x55641890d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 --2- 172.21.15.98:0/2561872197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5564188fd810 0x55641890cb70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f5467e00640 0 client.15735 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 >> 172.21.15.98:0/2561872197 conn(0x5564188f8d80 msgr2=0x5564188f7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 shutdown_connections 2024-03-16T12:23:23.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 1 -- 172.21.15.98:0/2561872197 wait complete. 2024-03-16T12:23:23.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.824+0000 7f548ff88740 20 client.15735 trim_cache size 0 max 16384 2024-03-16T12:23:23.812 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T12:23:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "49488d6a-af6b-4125-8a8c-42313ce21671": { 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "49488d6a-af6b-4125-8a8c-42313ce21671", 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:23:23.813 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 msgr2=0x7f546c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 0x7f546c0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5564188fea60 tx=0x7f5480009500 comp rx=0 tx=0).stop 2024-03-16T12:23:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 msgr2=0x7f5488199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f547802c650 comp rx=0 tx=0).stop 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 shutdown_connections 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f546c0a2800 0x7f546c0a4cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54881033a0 0x7f54881a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54881029e0 0x7f548819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 --2- 172.21.15.98:0/137814288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54881017f0 0x7f5488199d70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 >> 172.21.15.98:0/137814288 conn(0x7f54880fcd80 msgr2=0x7f5488073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 shutdown_connections 2024-03-16T12:23:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:23.825+0000 7f548d200640 1 -- 172.21.15.98:0/137814288 wait complete. 2024-03-16T12:23:23.822 INFO:tasks.cephfs.filesystem:scrub status for tag:49488d6a-af6b-4125-8a8c-42313ce21671 - {'path': '/', 'tag': '49488d6a-af6b-4125-8a8c-42313ce21671', 'options': 'recursive,force'} 2024-03-16T12:23:24.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:24 smithi173 ceph-mon[36632]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 777 KiB/s wr, 9 op/s 2024-03-16T12:23:24.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:24 smithi098 ceph-mon[31633]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 777 KiB/s wr, 9 op/s 2024-03-16T12:23:24.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:24 smithi153 ceph-mon[36901]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 777 KiB/s wr, 9 op/s 2024-03-16T12:23:25.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:25 smithi173 ceph-mon[36632]: scrub queued for path: / 2024-03-16T12:23:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:25 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:25 smithi173 ceph-mon[36632]: scrub summary: active paths [/] 2024-03-16T12:23:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:25 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:25 smithi173 ceph-mon[36632]: scrub summary: idle 2024-03-16T12:23:25.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:25 smithi098 ceph-mon[31633]: scrub queued for path: / 2024-03-16T12:23:25.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:25 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:25 smithi098 ceph-mon[31633]: scrub summary: active paths [/] 2024-03-16T12:23:25.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:25 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:25 smithi098 ceph-mon[31633]: scrub summary: idle 2024-03-16T12:23:25.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:25 smithi153 ceph-mon[36901]: scrub queued for path: / 2024-03-16T12:23:25.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:25 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:25 smithi153 ceph-mon[36901]: scrub summary: active paths [/] 2024-03-16T12:23:25.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:25 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:25.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:25 smithi153 ceph-mon[36901]: scrub summary: idle 2024-03-16T12:23:26.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:26 smithi173 ceph-mon[36632]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:26.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:26 smithi098 ceph-mon[31633]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:26.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:26 smithi153 ceph-mon[36901]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:27.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:27 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:23:27.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:27 smithi173 ceph-mon[36632]: pgmap v463: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-03-16T12:23:27.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:27 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:23:27.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:27 smithi098 ceph-mon[31633]: pgmap v463: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-03-16T12:23:27.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:27 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:23:27.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:27 smithi153 ceph-mon[36901]: pgmap v463: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-03-16T12:23:28.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:23:28.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:23:28.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:23:28.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:23:28.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:23:28.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:23:28.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:23:28.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:23:28.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:23:29.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:29 smithi173 ceph-mon[36632]: pgmap v464: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:29.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:29 smithi098 ceph-mon[31633]: pgmap v464: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:29.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:29 smithi153 ceph-mon[36901]: pgmap v464: 129 pgs: 129 active+clean; 1.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 773 KiB/s wr, 8 op/s 2024-03-16T12:23:32.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:31 smithi098 ceph-mon[31633]: pgmap v465: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:23:32.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:31 smithi153 ceph-mon[36901]: pgmap v465: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:23:32.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:31 smithi173 ceph-mon[36632]: pgmap v465: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:23:34.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:33 smithi098 ceph-mon[31633]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 9 op/s 2024-03-16T12:23:34.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:33 smithi153 ceph-mon[36901]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 9 op/s 2024-03-16T12:23:34.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:33 smithi173 ceph-mon[36632]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 9 op/s 2024-03-16T12:23:36.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:35 smithi098 ceph-mon[31633]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:23:36.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:35 smithi153 ceph-mon[36901]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:23:36.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:35 smithi173 ceph-mon[36632]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:23:38.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:37 smithi098 ceph-mon[31633]: pgmap v468: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:23:38.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:37 smithi153 ceph-mon[36901]: pgmap v468: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:23:38.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:37 smithi173 ceph-mon[36632]: pgmap v468: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:23:40.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:39 smithi098 ceph-mon[31633]: pgmap v469: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:23:40.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:39 smithi153 ceph-mon[36901]: pgmap v469: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:23:40.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:39 smithi173 ceph-mon[36632]: pgmap v469: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:23:42.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:41 smithi098 ceph-mon[31633]: pgmap v470: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:23:42.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:41 smithi153 ceph-mon[36901]: pgmap v470: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:23:42.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:41 smithi173 ceph-mon[36632]: pgmap v470: 129 pgs: 129 active+clean; 1.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:23:44.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:43 smithi098 ceph-mon[31633]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 9 op/s 2024-03-16T12:23:44.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:43 smithi153 ceph-mon[36901]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 9 op/s 2024-03-16T12:23:44.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:43 smithi173 ceph-mon[36632]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 9 op/s 2024-03-16T12:23:46.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:45 smithi098 ceph-mon[31633]: pgmap v472: 129 pgs: 129 active+clean; 114 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 12 op/s 2024-03-16T12:23:46.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:45 smithi153 ceph-mon[36901]: pgmap v472: 129 pgs: 129 active+clean; 114 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 12 op/s 2024-03-16T12:23:46.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:45 smithi173 ceph-mon[36632]: pgmap v472: 129 pgs: 129 active+clean; 114 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 10 MiB/s wr, 12 op/s 2024-03-16T12:23:48.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:47 smithi098 ceph-mon[31633]: pgmap v473: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 MiB/s wr, 18 op/s 2024-03-16T12:23:48.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:47 smithi153 ceph-mon[36901]: pgmap v473: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 MiB/s wr, 18 op/s 2024-03-16T12:23:48.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:47 smithi173 ceph-mon[36632]: pgmap v473: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 MiB/s wr, 18 op/s 2024-03-16T12:23:50.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:50 smithi098 ceph-mon[31633]: pgmap v474: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:50.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:50 smithi153 ceph-mon[36901]: pgmap v474: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:50.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:50 smithi173 ceph-mon[36632]: pgmap v474: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:52.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:52 smithi098 ceph-mon[31633]: pgmap v475: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 17 op/s 2024-03-16T12:23:52.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:52 smithi153 ceph-mon[36901]: pgmap v475: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 17 op/s 2024-03-16T12:23:52.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:52 smithi173 ceph-mon[36632]: pgmap v475: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 17 op/s 2024-03-16T12:23:53.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:23:53.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:23:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 -- 172.21.15.98:0/1210711136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0103a20 msgr2=0x7f1fb0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/1210711136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0103a20 0x7f1fb0103e40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009900 tx=0x7f1fa402bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 -- 172.21.15.98:0/1210711136 shutdown_connections 2024-03-16T12:23:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/1210711136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb01055d0 0x7f1fb010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/1210711136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb0104c10 0x7f1fb0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/1210711136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0103a20 0x7f1fb0103e40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 -- 172.21.15.98:0/1210711136 >> 172.21.15.98:0/1210711136 conn(0x7f1fb00fef90 msgr2=0x7f1fb01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 -- 172.21.15.98:0/1210711136 shutdown_connections 2024-03-16T12:23:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.060+0000 7f1fb7a00640 1 -- 172.21.15.98:0/1210711136 wait complete. 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 Processor -- start 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 -- start start 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb0103a20 0x7f1fb0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb01055d0 0x7f1fb01a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010f6e0 con 0x7f1fb0103a20 2024-03-16T12:23:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010f560 con 0x7f1fb01055d0 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb7a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010f860 con 0x7f1fb0104c10 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb0103a20 0x7f1fb0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.061+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55158/0 (socket says 172.21.15.98:55158) 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb01055d0 0x7f1fb01a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 -- 172.21.15.98:0/958511011 learned_addr learned my addr 172.21.15.98:0/958511011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 -- 172.21.15.98:0/958511011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb01055d0 msgr2=0x7f1fb01a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb01055d0 0x7f1fb01a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 -- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb0103a20 msgr2=0x7f1fb0199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb0103a20 0x7f1fb0199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 -- 172.21.15.98:0/958511011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb01a19b0 con 0x7f1fb0104c10 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fafe00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00e9f0 tx=0x7f1f9c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fad600640 1 -- 172.21.15.98:0/958511011 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c00cd80 con 0x7f1fb0104c10 2024-03-16T12:23:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fad600640 1 -- 172.21.15.98:0/958511011 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f9c005500 con 0x7f1fb0104c10 2024-03-16T12:23:54.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.062+0000 7f1fad600640 1 -- 172.21.15.98:0/958511011 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c010690 con 0x7f1fb0104c10 2024-03-16T12:23:54.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.063+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01a1ca0 con 0x7f1fb0104c10 2024-03-16T12:23:54.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.063+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb01a2170 con 0x7f1fb0104c10 2024-03-16T12:23:54.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.065+0000 7f1fad600640 1 -- 172.21.15.98:0/958511011 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1f9c0046c0 con 0x7f1fb0104c10 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 Processor -- start 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 -- start start 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a34d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a35370 0x5590f4a37800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590f49941f0 con 0x5590f4a35370 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590f493ca80 con 0x5590f4a2f810 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5590f4990290 con 0x5590f4a30200 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a34d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55430/0 (socket says 172.21.15.98:55430) 2024-03-16T12:23:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.066+0000 7f1fb5a00640 1 -- 172.21.15.98:0/85849698 learned_addr learned my addr 172.21.15.98:0/85849698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fafe00640 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a35370 0x5590f4a37800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 -- 172.21.15.98:0/85849698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 msgr2=0x5590f4a34d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a34d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 -- 172.21.15.98:0/85849698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a35370 msgr2=0x5590f4a37800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a35370 0x5590f4a37800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 -- 172.21.15.98:0/85849698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590f47727d0 con 0x5590f4a2f810 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fad600640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 0x7f1f980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5a00640 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=573b5a9416834adb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fad600640 1 -- 172.21.15.98:0/958511011 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1f9c0cf230 con 0x7f1fb0104c10 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fb5000640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 0x7f1f980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fa2000640 1 -- 172.21.15.98:0/85849698 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8004d20 con 0x5590f4a2f810 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fa2000640 1 -- 172.21.15.98:0/85849698 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa8004050 con 0x5590f4a2f810 2024-03-16T12:23:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.067+0000 7f1fa2000640 1 -- 172.21.15.98:0/85849698 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8005260 con 0x5590f4a2f810 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 -- 172.21.15.98:0/85849698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 msgr2=0x5590f4a2fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fb5000640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 0x7f1f980a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1fa402c170 tx=0x7f1fa402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa80118d0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 -- 172.21.15.98:0/85849698 shutdown_connections 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a35370 0x5590f4a37800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a34d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/85849698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a2fc30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 -- 172.21.15.98:0/85849698 >> 172.21.15.98:0/85849698 conn(0x5590f4a2ad80 msgr2=0x5590f4a2b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 -- 172.21.15.98:0/85849698 shutdown_connections 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.068+0000 7f1fba5fc740 1 -- 172.21.15.98:0/85849698 wait complete. 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 -- start start 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 Processor -- start 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a39d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a3a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 0x5590f4a41280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590f4a38320 con 0x5590f4a30200 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.069+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5590f494fed0 con 0x5590f4a2f810 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5590f4a384a0 con 0x5590f4a35370 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fba5fc740 10 client.0 _ll_register_callbacks cb 0x5590f498b220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 0x5590f4a41280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a39d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a3a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 0x5590f4a41280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55186/0 (socket says 172.21.15.98:55186) 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5a00640 1 -- 172.21.15.98:0/443295720 learned_addr learned my addr 172.21.15.98:0/443295720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 -- 172.21.15.98:0/443295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 msgr2=0x5590f4a41280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 0x5590f4a41280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 -- 172.21.15.98:0/443295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 msgr2=0x5590f4a3a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a3a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 -- 172.21.15.98:0/443295720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590f46a83f0 con 0x5590f4a2f810 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1fb5000640 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a39d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009230 tx=0x7f1fa4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb2dad64fafb9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.070+0000 7f1f96a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4045070 con 0x5590f4a2f810 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1fa40035f0 con 0x5590f4a2f810 2024-03-16T12:23:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa8004050 con 0x5590f4a2f810 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1fa40037f0 con 0x5590f4a2f810 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1fba5fc740 10 client.25910 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5590f46680f0 con 0x5590f4a2f810 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa8004050 con 0x5590f4a2f810 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.071+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1fa4034a30 con 0x5590f4a2f810 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 10 client.25910 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 10 client.25910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 20 client.25910 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4a45c00 0x5590f4a48070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 4 client.25910 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590f49a7be0 con 0x5590f4a45c00 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fafe00640 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4a45c00 0x5590f4a48070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1f96a00640 10 client.25910 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:54.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.072+0000 7f1fafe00640 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4a45c00 0x5590f4a48070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b97c5618be0279ae server_cookie=2e41924d6cd2a5a6 in_seq=0 out_seq=0 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1f96a00640 1 -- 172.21.15.98:0/443295720 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590f49a7be0 con 0x5590f4a45c00 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1f96a00640 10 client.25910 handle_command_reply: tid=0 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 client.25910 shutdown 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 msgr2=0x5590f4a39d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a39d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009230 tx=0x7f1fa4009260 comp rx=0 tx=0).stop 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 shutdown_connections 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4a45c00 0x5590f4a48070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a35370 0x5590f4a41280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a3a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1f96a00640 0 client.25910 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/443295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a2f810 0x5590f4a39d40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 >> 172.21.15.98:0/443295720 conn(0x5590f4a2ad80 msgr2=0x5590f4a29ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 shutdown_connections 2024-03-16T12:23:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.073+0000 7f1fba5fc740 1 -- 172.21.15.98:0/443295720 wait complete. 2024-03-16T12:23:54.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.074+0000 7f1fba5fc740 20 client.25910 trim_cache size 0 max 16384 2024-03-16T12:23:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.092+0000 7f1fba5fc740 1 Processor -- start 2024-03-16T12:23:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.092+0000 7f1fba5fc740 1 -- start start 2024-03-16T12:23:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a41240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a29630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01aedf0 con 0x5590f4a30200 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01ac6c0 con 0x5590f4a35370 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01aa510 con 0x5590f4a2f810 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a29630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55188/0 (socket says 172.21.15.98:55188) 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a29630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55458/0 (socket says 172.21.15.98:55458) 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 -- 172.21.15.98:0/1251282709 learned_addr learned my addr 172.21.15.98:0/1251282709 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fafe00640 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a41240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 -- 172.21.15.98:0/1251282709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 msgr2=0x5590f4a29630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a29630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 -- 172.21.15.98:0/1251282709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 msgr2=0x5590f4a41240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a41240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.093+0000 7f1fb5000640 1 -- 172.21.15.98:0/1251282709 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590f4a31de0 con 0x5590f4a2f810 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.094+0000 7f1fb5000640 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4034f20 tx=0x7f1fa402ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f952921d52ba20f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.094+0000 7f1fa1600640 1 -- 172.21.15.98:0/1251282709 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4067070 con 0x5590f4a2f810 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.094+0000 7f1fa1600640 1 -- 172.21.15.98:0/1251282709 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa4003e70 con 0x5590f4a2f810 2024-03-16T12:23:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.094+0000 7f1fa1600640 1 -- 172.21.15.98:0/1251282709 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4054550 con 0x5590f4a2f810 2024-03-16T12:23:54.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- 172.21.15.98:0/1251282709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 msgr2=0x5590f4a25d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4034f20 tx=0x7f1fa402ca00 comp rx=0 tx=0).stop 2024-03-16T12:23:54.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- 172.21.15.98:0/1251282709 shutdown_connections 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a29630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a30200 0x5590f4a41240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/1251282709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a2f810 0x5590f4a25d50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- 172.21.15.98:0/1251282709 >> 172.21.15.98:0/1251282709 conn(0x5590f4a2ad80 msgr2=0x5590f4a39ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- 172.21.15.98:0/1251282709 shutdown_connections 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- 172.21.15.98:0/1251282709 wait complete. 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 -- start start 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.095+0000 7f1fba5fc740 1 Processor -- start 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a3f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a40340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01aab80 con 0x5590f4a2f810 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01abff0 con 0x5590f4a35370 2024-03-16T12:23:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb01ab250 con 0x5590f4a30200 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fba5fc740 10 client.0 _ll_register_callbacks cb 0x7f1fa4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a3f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fb5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48174/0 (socket says 172.21.15.98:48174) 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fb5000640 1 -- 172.21.15.98:0/2007823728 learned_addr learned my addr 172.21.15.98:0/2007823728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fb5a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a40340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 -- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 msgr2=0x5590f4a40340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a40340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 -- 172.21.15.98:0/2007823728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 msgr2=0x5590f4a3ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fafe00640 1 -- 172.21.15.98:0/2007823728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590f4a40930 con 0x5590f4a30200 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.096+0000 7f1fb5000640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fafe00640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a3f380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0cc970 tx=0x7f1f9c0cc840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8e0ccf4b9e00cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c09b810 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1f9c0860b0 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa4003e70 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1f9c09f960 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 10 client.25906 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5590f4aac4a0 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa4003e70 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1f9c097040 con 0x5590f4a30200 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 10 client.25906 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 20 client.25906 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4aace30 0x5590f4aaf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 4 client.25906 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:23:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.097+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590f49a1730 con 0x5590f4aace30 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1fb5000640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4aace30 0x5590f4aaf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1fb5000640 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4aace30 0x5590f4aaf220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9d5b01ef0e7815c server_cookie=acf2d0763edf62b3 in_seq=0 out_seq=0 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1f96a00640 10 client.25906 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1f96a00640 1 -- 172.21.15.98:0/2007823728 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5590f49a1730 con 0x5590f4aace30 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1f96a00640 10 client.25906 handle_command_reply: tid=0 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1fba5fc740 1 client.25906 shutdown 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 msgr2=0x5590f4a3f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.098+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a3f380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0cc970 tx=0x7f1f9c0cc840 comp rx=0 tx=0).stop 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 shutdown_connections 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5590f4a35370 0x5590f4a40340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x5590f4aace30 0x5590f4aaf220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5590f4a30200 0x5590f4a3f380 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 --2- 172.21.15.98:0/2007823728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5590f4a2f810 0x5590f4a3ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1f96a00640 0 client.25906 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 >> 172.21.15.98:0/2007823728 conn(0x5590f4a2ad80 msgr2=0x5590f4a29630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 shutdown_connections 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 1 -- 172.21.15.98:0/2007823728 wait complete. 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fba5fc740 20 client.25906 trim_cache size 0 max 16384 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-03-16T12:23:54.088 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 msgr2=0x7f1f980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 0x7f1f980a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1fa402c170 tx=0x7f1fa402df90 comp rx=0 tx=0).stop 2024-03-16T12:23:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 msgr2=0x7f1fb019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.099+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00e9f0 tx=0x7f1f9c00edf0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 shutdown_connections 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f1f980a2880 0x7f1f980a4d40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fb01055d0 0x7f1fb01a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fb0104c10 0x7f1fb019a2e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 --2- 172.21.15.98:0/958511011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fb0103a20 0x7f1fb0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 >> 172.21.15.98:0/958511011 conn(0x7f1fb00fef90 msgr2=0x7f1fb0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 shutdown_connections 2024-03-16T12:23:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.100+0000 7f1fb7a00640 1 -- 172.21.15.98:0/958511011 wait complete. 2024-03-16T12:23:54.096 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T12:23:54.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:23:54.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:54 smithi098 ceph-mon[31633]: pgmap v476: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-16T12:23:54.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:54 smithi153 ceph-mon[36901]: pgmap v476: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 -- 172.21.15.98:0/2900371922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc1020d0 msgr2=0x7fd5bc1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/2900371922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc1044c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009900 tx=0x7fd5ac02bc50 comp rx=0 tx=0).stop 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 -- 172.21.15.98:0/2900371922 shutdown_connections 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/2900371922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc107330 0x7fd5bc109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/2900371922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc104a00 0x7fd5bc106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/2900371922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc1044c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 -- 172.21.15.98:0/2900371922 >> 172.21.15.98:0/2900371922 conn(0x7fd5bc0fba60 msgr2=0x7fd5bc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 -- 172.21.15.98:0/2900371922 shutdown_connections 2024-03-16T12:23:54.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.316+0000 7fd5bbe00640 1 -- 172.21.15.98:0/2900371922 wait complete. 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.317+0000 7fd5bbe00640 1 Processor -- start 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 -- start start 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc104a00 0x7fd5bc197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc107330 0x7fd5bc19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc10d2c0 con 0x7fd5bc104a00 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc10d140 con 0x7fd5bc1020d0 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bbe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc10d440 con 0x7fd5bc107330 2024-03-16T12:23:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5b3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc104a00 0x7fd5bc197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55478/0 (socket says 172.21.15.98:55478) 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5baa00640 1 -- 172.21.15.98:0/160214841 learned_addr learned my addr 172.21.15.98:0/160214841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.318+0000 7fd5bb400640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc107330 0x7fd5bc19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 -- 172.21.15.98:0/160214841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc107330 msgr2=0x7fd5bc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc107330 0x7fd5bc19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 -- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc104a00 msgr2=0x7fd5bc197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc104a00 0x7fd5bc197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 -- 172.21.15.98:0/160214841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc19f520 con 0x7fd5bc1020d0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5baa00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac02c260 tx=0x7fd5ac02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac041070 con 0x7fd5bc1020d0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc19f7b0 con 0x7fd5bc1020d0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ac004450 con 0x7fd5bc1020d0 2024-03-16T12:23:54.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.319+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac003e50 con 0x7fd5bc1020d0 2024-03-16T12:23:54.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.320+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5bc19fd50 con 0x7fd5bc1020d0 2024-03-16T12:23:54.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.321+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5bc0fd550 con 0x7fd5bc1020d0 2024-03-16T12:23:54.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.322+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd5ac044070 con 0x7fd5bc1020d0 2024-03-16T12:23:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.324+0000 7fd5b3400640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 0x7fd5a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.324+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5ac0f0fb0 con 0x7fd5bc1020d0 2024-03-16T12:23:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.324+0000 7fd5b3e00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 0x7fd5a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.325+0000 7fd5b3e00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 0x7fd5a00a4e60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc198e10 tx=0x7fd5a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.329+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5ac0bdc90 con 0x7fd5bc1020d0 2024-03-16T12:23:54.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:54 smithi173 ceph-mon[36632]: pgmap v476: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-16T12:23:54.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.566+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5bc02d1b0 con 0x7fd5bc1020d0 2024-03-16T12:23:54.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.567+0000 7fd5b3400640 1 -- 172.21.15.98:0/160214841 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7fd5ac0bd590 con 0x7fd5bc1020d0 2024-03-16T12:23:54.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:23:54.548 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:23:54.548 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 msgr2=0x7fd5a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 0x7fd5a00a4e60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc198e10 tx=0x7fd5a4005f90 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 msgr2=0x7fd5bc197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac02c260 tx=0x7fd5ac02c670 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 shutdown_connections 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd5a00a29a0 0x7fd5a00a4e60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5bc107330 0x7fd5bc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5bc104a00 0x7fd5bc197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 --2- 172.21.15.98:0/160214841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5bc1020d0 0x7fd5bc197900 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 >> 172.21.15.98:0/160214841 conn(0x7fd5bc0fba60 msgr2=0x7fd5bc102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 shutdown_connections 2024-03-16T12:23:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.570+0000 7fd5bbe00640 1 -- 172.21.15.98:0/160214841 wait complete. 2024-03-16T12:23:54.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 -- 172.21.15.98:0/4186074455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc104be0 msgr2=0x7f74cc105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 --2- 172.21.15.98:0/4186074455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc104be0 0x7f74cc105060 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009900 tx=0x7f74b802bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 -- 172.21.15.98:0/4186074455 shutdown_connections 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 --2- 172.21.15.98:0/4186074455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc1055a0 0x7f74cc10be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 --2- 172.21.15.98:0/4186074455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc104be0 0x7f74cc105060 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 --2- 172.21.15.98:0/4186074455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1039f0 0x7f74cc103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 -- 172.21.15.98:0/4186074455 >> 172.21.15.98:0/4186074455 conn(0x7f74cc0fefa0 msgr2=0x7f74cc1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 -- 172.21.15.98:0/4186074455 shutdown_connections 2024-03-16T12:23:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.787+0000 7f74d1e00640 1 -- 172.21.15.98:0/4186074455 wait complete. 2024-03-16T12:23:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.788+0000 7f74d1e00640 1 Processor -- start 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.788+0000 7f74d1e00640 1 -- start start 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 0x7f74cc199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 0x7f74cc19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc10f6a0 con 0x7f74cc104be0 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc10f520 con 0x7f74cc1055a0 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74d1e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc10f820 con 0x7f74cc1039f0 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 0x7f74cc199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55506/0 (socket says 172.21.15.98:55506) 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 -- 172.21.15.98:0/2128187602 learned_addr learned my addr 172.21.15.98:0/2128187602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 0x7f74cc199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55212/0 (socket says 172.21.15.98:55212) 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74ca000640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 0x7f74cc19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 -- 172.21.15.98:0/2128187602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 msgr2=0x7f74cc199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 0x7f74cc199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 -- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 msgr2=0x7f74cc19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 0x7f74cc19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74cb400640 1 -- 172.21.15.98:0/2128187602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc1a18d0 con 0x7f74cc1055a0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.789+0000 7f74ca000640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 0x7f74cc19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74cb400640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f74c400eb10 tx=0x7f74c400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74d0a00640 1 -- 172.21.15.98:0/2128187602 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4004b20 con 0x7f74cc1055a0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc1a1bc0 con 0x7f74cc1055a0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74d0a00640 1 -- 172.21.15.98:0/2128187602 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c4004cc0 con 0x7f74cc1055a0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74d0a00640 1 -- 172.21.15.98:0/2128187602 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4010760 con 0x7f74cc1055a0 2024-03-16T12:23:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.790+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc1a20e0 con 0x7f74cc1055a0 2024-03-16T12:23:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.793+0000 7f74d4bad740 1 Processor -- start 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.793+0000 7f74d0a00640 1 -- 172.21.15.98:0/2128187602 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f74c4005020 con 0x7f74cc1055a0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.793+0000 7f74d4bad740 1 -- start start 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e6dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559179e353b0 con 0x559179e73420 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559179dd2450 con 0x559179e6d8c0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74d4bad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559179d7acb0 con 0x559179e6e2b0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55228/0 (socket says 172.21.15.98:55228) 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 -- 172.21.15.98:0/2137350376 learned_addr learned my addr 172.21.15.98:0/2137350376 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74ca000640 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e6dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74ca000640 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 -- 172.21.15.98:0/2137350376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 msgr2=0x559179e6dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e6dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 -- 172.21.15.98:0/2137350376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 msgr2=0x559179e758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e758b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.794+0000 7f74caa00640 1 -- 172.21.15.98:0/2137350376 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559179d9c1f0 con 0x559179e6e2b0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74caa00640 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009c60 tx=0x7f74bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=543c0c8193c33eef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74b7e00640 1 -- 172.21.15.98:0/2137350376 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc004220 con 0x559179e6e2b0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74d0a00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 0x7f74b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74b7e00640 1 -- 172.21.15.98:0/2137350376 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74bc0043c0 con 0x559179e6e2b0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74b7e00640 1 -- 172.21.15.98:0/2137350376 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc004d50 con 0x559179e6e2b0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74d0a00640 1 -- 172.21.15.98:0/2128187602 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74c40cf3a0 con 0x7f74cc1055a0 2024-03-16T12:23:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.795+0000 7f74ca000640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 0x7f74b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- 172.21.15.98:0/2137350376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 msgr2=0x559179e72db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009c60 tx=0x7f74bc011810 comp rx=0 tx=0).stop 2024-03-16T12:23:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- 172.21.15.98:0/2137350376 shutdown_connections 2024-03-16T12:23:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e72db0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 --2- 172.21.15.98:0/2137350376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e6dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- 172.21.15.98:0/2137350376 >> 172.21.15.98:0/2137350376 conn(0x559179e68e30 msgr2=0x559179e69230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- 172.21.15.98:0/2137350376 shutdown_connections 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74ca000640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 0x7f74b00a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f74b802c240 tx=0x7f74b802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- 172.21.15.98:0/2137350376 wait complete. 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.796+0000 7f74d4bad740 1 -- start start 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 Processor -- start 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6e2b0 0x559179e78050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e7f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559179d8e110 con 0x559179e73420 2024-03-16T12:23:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559179dce4f0 con 0x559179e6e2b0 2024-03-16T12:23:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559179e763d0 con 0x559179e6d8c0 2024-03-16T12:23:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74d4bad740 10 client.0 _ll_register_callbacks cb 0x559179d7bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74caa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55236/0 (socket says 172.21.15.98:55236) 2024-03-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74caa00640 1 -- 172.21.15.98:0/4225224507 learned_addr learned my addr 172.21.15.98:0/4225224507 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74ca000640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6e2b0 0x559179e78050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.797+0000 7f74cb400640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e7f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 -- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6e2b0 msgr2=0x559179e78050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6e2b0 0x559179e78050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 -- 172.21.15.98:0/4225224507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 msgr2=0x559179e7f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e7f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 -- 172.21.15.98:0/4225224507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559179ae63f0 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74caa00640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f74bc011d10 tx=0x7f74bc0052d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79bb94f12d72676c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc027030 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74bc0043c0 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.798+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74bc019430 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f74bc034050 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 10 client.25924 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559179bb07d0 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74bc0043c0 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f74bc019cb0 con 0x559179e6d8c0 2024-03-16T12:23:54.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 10 client.25924 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 10 client.25924 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 20 client.25924 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179e83760 0x559179e85bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 4 client.25924 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559179ddf990 con 0x559179e83760 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.799+0000 7f74ca000640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179e83760 0x559179e85bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.800+0000 7f74ca000640 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179e83760 0x559179e85bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0090e36a4e754b9 server_cookie=1dc31fa9739b43ad in_seq=0 out_seq=0 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.800+0000 7f74b4c00640 10 client.25924 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74b4c00640 1 -- 172.21.15.98:0/4225224507 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559179ddf990 con 0x559179e83760 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74b4c00640 10 client.25924 handle_command_reply: tid=0 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 client.25924 shutdown 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 msgr2=0x559179e77b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f74bc011d10 tx=0x7f74bc0052d0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 shutdown_connections 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179e83760 0x559179e85bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e73420 0x559179e7f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6e2b0 0x559179e78050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74b4c00640 0 client.25924 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4225224507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6d8c0 0x559179e77b10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 >> 172.21.15.98:0/4225224507 conn(0x559179e68e30 msgr2=0x559179e681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 shutdown_connections 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 1 -- 172.21.15.98:0/4225224507 wait complete. 2024-03-16T12:23:54.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.801+0000 7f74d4bad740 20 client.25924 trim_cache size 0 max 16384 2024-03-16T12:23:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 Processor -- start 2024-03-16T12:23:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 -- start start 2024-03-16T12:23:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6d8c0 0x559179e6c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 0x559179e767a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1aed60 con 0x559179e6d8c0 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1ac630 con 0x559179e73420 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.816+0000 7f74d4bad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1aa480 con 0x559179e6e2b0 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 0x559179e767a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74caa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6d8c0 0x559179e6c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55248/0 (socket says 172.21.15.98:55248) 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 0x559179e767a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55536/0 (socket says 172.21.15.98:55536) 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 -- 172.21.15.98:0/4108657740 learned_addr learned my addr 172.21.15.98:0/4108657740 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 -- 172.21.15.98:0/4108657740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 msgr2=0x559179e767a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 0x559179e767a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 -- 172.21.15.98:0/4108657740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6d8c0 msgr2=0x559179e6c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6d8c0 0x559179e6c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 -- 172.21.15.98:0/4108657740 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559179e69ba0 con 0x559179e6e2b0 2024-03-16T12:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74ca000640 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f74b802c7b0 tx=0x7f74b803a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea30d398aea5819 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74b7400640 1 -- 172.21.15.98:0/4108657740 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8056030 con 0x559179e6e2b0 2024-03-16T12:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74b7400640 1 -- 172.21.15.98:0/4108657740 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b80056a0 con 0x559179e6e2b0 2024-03-16T12:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.817+0000 7f74b7400640 1 -- 172.21.15.98:0/4108657740 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8004920 con 0x559179e6e2b0 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 -- 172.21.15.98:0/4108657740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 msgr2=0x559179e8b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f74b802c7b0 tx=0x7f74b803a000 comp rx=0 tx=0).stop 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 -- 172.21.15.98:0/4108657740 shutdown_connections 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e73420 0x559179e767a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e6e2b0 0x559179e8b4a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 --2- 172.21.15.98:0/4108657740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6d8c0 0x559179e6c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 -- 172.21.15.98:0/4108657740 >> 172.21.15.98:0/4108657740 conn(0x559179e68e30 msgr2=0x559179e6c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 -- 172.21.15.98:0/4108657740 shutdown_connections 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.818+0000 7f74d4bad740 1 -- 172.21.15.98:0/4108657740 wait complete. 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 -- start start 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 Processor -- start 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e7c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 0x559179e7c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 0x559179e7d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1aaaf0 con 0x559179e6e2b0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1abf60 con 0x559179e6d8c0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1ab1c0 con 0x559179e73420 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.819+0000 7f74d4bad740 10 client.0 _ll_register_callbacks cb 0x7f74bc009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74caa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e7c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 0x559179e7c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74caa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e7c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74caa00640 1 -- 172.21.15.98:0/761397185 learned_addr learned my addr 172.21.15.98:0/761397185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74cb400640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 0x559179e7d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 -- 172.21.15.98:0/761397185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 msgr2=0x559179e7d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 0x559179e7d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 -- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 msgr2=0x559179e7c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e7c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 -- 172.21.15.98:0/761397185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559179e7dc20 con 0x559179e6e2b0 2024-03-16T12:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74cb400640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 0x559179e7d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74ca000640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 0x559179e7c620 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74b8036040 tx=0x7f74b8052890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8ba6224a649f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b805f070 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74b80056a0 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.820+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74b8052070 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f74b806b050 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 10 client.15777 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559179e7ef20 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b80056a0 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f74b8040040 con 0x559179e6e2b0 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 10 client.15777 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 10 client.15777 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 20 client.15777 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179eea0a0 0x559179eec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 4 client.15777 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559179dd9ce0 con 0x559179eea0a0 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.821+0000 7f74caa00640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179eea0a0 0x559179eec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74caa00640 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179eea0a0 0x559179eec4c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57789d40015694d2 server_cookie=4caf1bff45f2e9f7 in_seq=0 out_seq=0 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74b4c00640 10 client.15777 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74b4c00640 1 -- 172.21.15.98:0/761397185 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559179dd9ce0 con 0x559179eea0a0 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74b4c00640 10 client.15777 handle_command_reply: tid=0 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 client.15777 shutdown 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 msgr2=0x559179e7c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 0x559179e7c620 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74b8036040 tx=0x7f74b8052890 comp rx=0 tx=0).stop 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 shutdown_connections 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x559179eea0a0 0x559179eec4c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559179e73420 0x559179e7d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559179e6e2b0 0x559179e7c620 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 --2- 172.21.15.98:0/761397185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559179e6d8c0 0x559179e7c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 >> 172.21.15.98:0/761397185 conn(0x559179e68e30 msgr2=0x559179e6c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74b4c00640 0 client.15777 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.822+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 shutdown_connections 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d4bad740 1 -- 172.21.15.98:0/761397185 wait complete. 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d4bad740 20 client.15777 trim_cache size 0 max 16384 2024-03-16T12:23:54.808 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 msgr2=0x7f74b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 0x7f74b00a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f74b802c240 tx=0x7f74b802df90 comp rx=0 tx=0).stop 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 msgr2=0x7f74cc1a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f74c400eb10 tx=0x7f74c400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 shutdown_connections 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f74b00a2800 0x7f74b00a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74cc1055a0 0x7f74cc1a1230 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74cc104be0 0x7f74cc19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 --2- 172.21.15.98:0/2128187602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74cc1039f0 0x7f74cc199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.823+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 >> 172.21.15.98:0/2128187602 conn(0x7f74cc0fefa0 msgr2=0x7f74cc107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.824+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 shutdown_connections 2024-03-16T12:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:54.824+0000 7f74d1e00640 1 -- 172.21.15.98:0/2128187602 wait complete. 2024-03-16T12:23:54.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T12:23:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 -- 172.21.15.98:0/2206148367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100773e0 msgr2=0x7f2a10077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 --2- 172.21.15.98:0/2206148367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100773e0 0x7f2a10077840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009900 tx=0x7f29fc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:23:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 -- 172.21.15.98:0/2206148367 shutdown_connections 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 --2- 172.21.15.98:0/2206148367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10077d80 0x7f2a10078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 --2- 172.21.15.98:0/2206148367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100773e0 0x7f2a10077840 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 --2- 172.21.15.98:0/2206148367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a100765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 -- 172.21.15.98:0/2206148367 >> 172.21.15.98:0/2206148367 conn(0x7f2a100fef90 msgr2=0x7f2a101013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 -- 172.21.15.98:0/2206148367 shutdown_connections 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.042+0000 7f2a15000640 1 -- 172.21.15.98:0/2206148367 wait complete. 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.043+0000 7f2a15000640 1 Processor -- start 2024-03-16T12:23:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 -- start start 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a1019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10113ad0 con 0x7f2a100761a0 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10113950 con 0x7f2a10077d80 2024-03-16T12:23:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a15000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10113c50 con 0x7f2a100773e0 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a1019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55280/0 (socket says 172.21.15.98:55280) 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55566/0 (socket says 172.21.15.98:55566) 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0d400640 1 -- 172.21.15.98:0/2889273605 learned_addr learned my addr 172.21.15.98:0/2889273605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.044+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a1019e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48260/0 (socket says 172.21.15.98:48260) 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 -- 172.21.15.98:0/2889273605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 msgr2=0x7f2a101a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 -- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 msgr2=0x7f2a1019e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a1019e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 -- 172.21.15.98:0/2889273605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a101a5cf0 con 0x7f2a100773e0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0e800640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a0d400640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a06a00640 1 -- 172.21.15.98:0/2889273605 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc043020 con 0x7f2a100773e0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a06a00640 1 -- 172.21.15.98:0/2889273605 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29fc0054e0 con 0x7f2a100773e0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a06a00640 1 -- 172.21.15.98:0/2889273605 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc003c40 con 0x7f2a100773e0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a101a5f80 con 0x7f2a100773e0 2024-03-16T12:23:55.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.045+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a101a63f0 con 0x7f2a100773e0 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.048+0000 7f2a06a00640 1 -- 172.21.15.98:0/2889273605 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f29fc046020 con 0x7f2a100773e0 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.048+0000 7f2a17d91740 1 Processor -- start 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 -- start start 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112005db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a1120088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a111fc83b0 con 0x55a1120012b0 2024-03-16T12:23:55.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a111f65450 con 0x55a112006420 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a17d91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a111f0dcb0 con 0x55a1120008c0 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55284/0 (socket says 172.21.15.98:55284) 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 -- 172.21.15.98:0/1627496491 learned_addr learned my addr 172.21.15.98:0/1627496491 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0de00640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112005db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0d400640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a1120088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 -- 172.21.15.98:0/1627496491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 msgr2=0x55a1120088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a1120088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 -- 172.21.15.98:0/1627496491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 msgr2=0x55a112005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.049+0000 7f2a0e800640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112005db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a0e800640 1 -- 172.21.15.98:0/1627496491 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a111f2f1f0 con 0x55a1120008c0 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a0de00640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112005db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a06a00640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 0x7f29f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a0e800640 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2a080117e0 tx=0x7f2a08011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d347a2ecbc7cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a06a00640 1 -- 172.21.15.98:0/2889273605 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29fc0f0de0 con 0x7f2a100773e0 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f2a0de00640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 0x7f29f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f29fb400640 1 -- 172.21.15.98:0/1627496491 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08011310 con 0x55a1120008c0 2024-03-16T12:23:55.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f29fb400640 1 -- 172.21.15.98:0/1627496491 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a08003e50 con 0x55a1120008c0 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.050+0000 7f29fb400640 1 -- 172.21.15.98:0/1627496491 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a080045b0 con 0x55a1120008c0 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- 172.21.15.98:0/1627496491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 msgr2=0x55a112000ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a0de00640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 0x7f29f40a4d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55a1120016b0 tx=0x7f2a00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2a080117e0 tx=0x7f2a08011ca0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- 172.21.15.98:0/1627496491 shutdown_connections 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a1120088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112005db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 --2- 172.21.15.98:0/1627496491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a112000ce0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- 172.21.15.98:0/1627496491 >> 172.21.15.98:0/1627496491 conn(0x55a111ffbe30 msgr2=0x55a111ffc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- 172.21.15.98:0/1627496491 shutdown_connections 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- 172.21.15.98:0/1627496491 wait complete. 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.051+0000 7f2a17d91740 1 -- start start 2024-03-16T12:23:55.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 Processor -- start 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112012180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a111f21110 con 0x55a112006420 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a111f614f0 con 0x55a1120012b0 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a1120093d0 con 0x55a1120008c0 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a17d91740 10 client.0 _ll_register_callbacks cb 0x55a111f0eb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55298/0 (socket says 172.21.15.98:55298) 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55586/0 (socket says 172.21.15.98:55586) 2024-03-16T12:23:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0de00640 1 -- 172.21.15.98:0/3645859602 learned_addr learned my addr 172.21.15.98:0/3645859602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.052+0000 7f2a0e800640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112012180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 -- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 msgr2=0x55a11200ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 -- 172.21.15.98:0/3645859602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 msgr2=0x55a112012180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112012180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 -- 172.21.15.98:0/3645859602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a111c793f0 con 0x55a1120012b0 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0de00640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a0d400640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f29fc006010 tx=0x7f29fc0bd250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4792656e9a70495 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f29efe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:55.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0035f0 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29fc0bb6c0 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a08003e50 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.053+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f29fc005690 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 10 client.25952 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a111d437d0 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a08003e50 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f29fc0d60d0 con 0x55a1120012b0 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 10 client.25952 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 10 client.25952 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 20 client.25952 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a112016960 0x55a112018dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 4 client.25952 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a111f72990 con 0x55a112016960 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.054+0000 7f2a0de00640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a112016960 0x55a112018dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.055+0000 7f2a0de00640 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a112016960 0x55a112018dc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2dd0a61d037eafd server_cookie=1832976162170128 in_seq=0 out_seq=0 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.055+0000 7f29efe00640 10 client.25952 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.055+0000 7f29efe00640 1 -- 172.21.15.98:0/3645859602 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a111f72990 con 0x55a112016960 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.055+0000 7f29efe00640 10 client.25952 handle_command_reply: tid=0 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 client.25952 shutdown 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 msgr2=0x55a11200b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f29fc006010 tx=0x7f29fc0bd250 comp rx=0 tx=0).stop 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 shutdown_connections 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a112016960 0x55a112018dc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112012180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200b140 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f29efe00640 0 client.25952 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 --2- 172.21.15.98:0/3645859602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 >> 172.21.15.98:0/3645859602 conn(0x55a111ffbe30 msgr2=0x55a111ffb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 shutdown_connections 2024-03-16T12:23:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 1 -- 172.21.15.98:0/3645859602 wait complete. 2024-03-16T12:23:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.056+0000 7f2a17d91740 20 client.25952 trim_cache size 0 max 16384 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 Processor -- start 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 -- start start 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112012140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101b3060 con 0x55a1120012b0 2024-03-16T12:23:55.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101b0930 con 0x55a112006420 2024-03-16T12:23:55.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a17d91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101ae780 con 0x55a1120008c0 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a0e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55598/0 (socket says 172.21.15.98:55598) 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.071+0000 7f2a0e800640 1 -- 172.21.15.98:0/2581754302 learned_addr learned my addr 172.21.15.98:0/2581754302 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 -- 172.21.15.98:0/2581754302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 msgr2=0x55a11200ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 -- 172.21.15.98:0/2581754302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 msgr2=0x55a112012140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0d400640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112012140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112012140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 -- 172.21.15.98:0/2581754302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a111ff6670 con 0x55a112006420 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0de00640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0d400640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112012140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.072+0000 7f2a0e800640 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a080117e0 tx=0x7f2a08005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a5f0e1963b2d53a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.073+0000 7f29faa00640 1 -- 172.21.15.98:0/2581754302 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0802a070 con 0x55a112006420 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.073+0000 7f29faa00640 1 -- 172.21.15.98:0/2581754302 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a08003e50 con 0x55a112006420 2024-03-16T12:23:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.073+0000 7f29faa00640 1 -- 172.21.15.98:0/2581754302 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08005920 con 0x55a112006420 2024-03-16T12:23:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.073+0000 7f2a17d91740 1 -- 172.21.15.98:0/2581754302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 msgr2=0x55a11200b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.073+0000 7f2a17d91740 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a080117e0 tx=0x7f2a08005bd0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 -- 172.21.15.98:0/2581754302 shutdown_connections 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a112006420 0x55a11200b140 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a1120012b0 0x55a112012140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 --2- 172.21.15.98:0/2581754302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 -- 172.21.15.98:0/2581754302 >> 172.21.15.98:0/2581754302 conn(0x55a111ffbe30 msgr2=0x55a111ff7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 -- 172.21.15.98:0/2581754302 shutdown_connections 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 -- 172.21.15.98:0/2581754302 wait complete. 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 -- start start 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 Processor -- start 2024-03-16T12:23:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.074+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a17d91740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112010f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a17d91740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101aedf0 con 0x55a112006420 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a17d91740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101b0260 con 0x55a1120012b0 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a17d91740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2a101af4c0 con 0x55a1120008c0 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a17d91740 10 client.0 _ll_register_callbacks cb 0x7f29fc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55322/0 (socket says 172.21.15.98:55322) 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 -- 172.21.15.98:0/235171034 learned_addr learned my addr 172.21.15.98:0/235171034 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0e800640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112010f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 -- 172.21.15.98:0/235171034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 msgr2=0x55a11200ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 -- 172.21.15.98:0/235171034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 msgr2=0x55a112010f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112010f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 -- 172.21.15.98:0/235171034 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a112011510 con 0x55a1120008c0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0d400640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0e800640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112010f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f2a0de00640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009730 tx=0x7f2a000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114c75bf205b5b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.075+0000 7f29efe00640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00035070 con 0x55a1120008c0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2a0002e470 con 0x55a1120008c0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a08003e50 con 0x55a1120008c0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2a00041050 con 0x55a1120008c0 2024-03-16T12:23:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 10 client.25948 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a11207cea0 con 0x55a1120008c0 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a08003e50 con 0x55a1120008c0 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2a0002c400 con 0x55a1120008c0 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 10 client.25948 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 10 client.25948 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 20 client.25948 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a11207d880 0x55a11207fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 4 client.25948 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.076+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a111f6cce0 con 0x55a11207d880 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f2a0d400640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a11207d880 0x55a11207fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f2a0d400640 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a11207d880 0x55a11207fca0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db16e25d0a76771a server_cookie=c9eed49695d5d6a0 in_seq=0 out_seq=0 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f29efe00640 10 client.25948 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f29efe00640 1 -- 172.21.15.98:0/235171034 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a111f6cce0 con 0x55a11207d880 2024-03-16T12:23:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f29efe00640 10 client.25948 handle_command_reply: tid=0 2024-03-16T12:23:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f2a17d91740 1 client.25948 shutdown 2024-03-16T12:23:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 msgr2=0x55a11200fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.077+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009730 tx=0x7f2a000046c0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 shutdown_connections 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x55a11207d880 0x55a11207fca0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a112006420 0x55a112010f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a1120012b0 0x55a11200ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 --2- 172.21.15.98:0/235171034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a1120008c0 0x55a11200fa20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 >> 172.21.15.98:0/235171034 conn(0x55a111ffbe30 msgr2=0x55a11200abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f29efe00640 0 client.25948 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:23:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 shutdown_connections 2024-03-16T12:23:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 1 -- 172.21.15.98:0/235171034 wait complete. 2024-03-16T12:23:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.078+0000 7f2a17d91740 20 client.25948 trim_cache size 0 max 16384 2024-03-16T12:23:55.066 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 msgr2=0x7f29f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 0x7f29f40a4d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55a1120016b0 tx=0x7f2a00005f90 comp rx=0 tx=0).stop 2024-03-16T12:23:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 msgr2=0x7f2a1019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc02c650 comp rx=0 tx=0).stop 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 shutdown_connections 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f29f40a2880 0x7f29f40a4d40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10077d80 0x7f2a101a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a100773e0 0x7f2a1019e660 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 --2- 172.21.15.98:0/2889273605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a100761a0 0x7f2a1019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 >> 172.21.15.98:0/2889273605 conn(0x7f2a100fef90 msgr2=0x7f2a10100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 shutdown_connections 2024-03-16T12:23:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.079+0000 7f2a15000640 1 -- 172.21.15.98:0/2889273605 wait complete. 2024-03-16T12:23:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:55 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/160214841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:55.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T12:23:55.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:55 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/160214841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:55.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 -- 172.21.15.98:0/4173590310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 msgr2=0x7f96b410bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 --2- 172.21.15.98:0/4173590310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b410bb20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009ca0 tx=0x7f96ac02c180 comp rx=0 tx=0).stop 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 -- 172.21.15.98:0/4173590310 shutdown_connections 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 --2- 172.21.15.98:0/4173590310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b410bb20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 --2- 172.21.15.98:0/4173590310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40741c0 0x7f96b4074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 --2- 172.21.15.98:0/4173590310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40757a0 0x7f96b4073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 -- 172.21.15.98:0/4173590310 >> 172.21.15.98:0/4173590310 conn(0x7f96b40fcd50 msgr2=0x7f96b40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 -- 172.21.15.98:0/4173590310 shutdown_connections 2024-03-16T12:23:55.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.307+0000 7f96bb600640 1 -- 172.21.15.98:0/4173590310 wait complete. 2024-03-16T12:23:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 Processor -- start 2024-03-16T12:23:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 -- start start 2024-03-16T12:23:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40741c0 0x7f96b4199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 0x7f96b419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b410f670 con 0x7f96b40741c0 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b410f4f0 con 0x7f96b4109690 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.308+0000 7f96bb600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96b410f7f0 con 0x7f96b40757a0 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40741c0 0x7f96b4199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 0x7f96b419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 0x7f96b419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55342/0 (socket says 172.21.15.98:55342) 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b3e00640 1 -- 172.21.15.98:0/1783173242 learned_addr learned my addr 172.21.15.98:0/1783173242 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 -- 172.21.15.98:0/1783173242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 msgr2=0x7f96b419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 0x7f96b419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 -- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40741c0 msgr2=0x7f96b4199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40741c0 0x7f96b4199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 -- 172.21.15.98:0/1783173242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b41a1900 con 0x7f96b4109690 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.309+0000 7f96b9600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b41a1260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96ac037040 tx=0x7f96ac0093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.310+0000 7f96b1600640 1 -- 172.21.15.98:0/1783173242 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac039040 con 0x7f96b4109690 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.310+0000 7f96b1600640 1 -- 172.21.15.98:0/1783173242 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96ac044070 con 0x7f96b4109690 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.310+0000 7f96b1600640 1 -- 172.21.15.98:0/1783173242 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0042a0 con 0x7f96b4109690 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.310+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b41a1b90 con 0x7f96b4109690 2024-03-16T12:23:55.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.310+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b41a20a0 con 0x7f96b4109690 2024-03-16T12:23:55.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 Processor -- start 2024-03-16T12:23:55.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96b1600640 1 -- 172.21.15.98:0/1783173242 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f96ac004500 con 0x7f96b4109690 2024-03-16T12:23:55.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 -- start start 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42cf8c0 0x561cb42cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42d78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561cb42973b0 con 0x561cb42d5420 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561cb4234450 con 0x561cb42cf8c0 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.313+0000 7f96be23e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561cb41dccb0 con 0x561cb42d02b0 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55348/0 (socket says 172.21.15.98:55348) 2024-03-16T12:23:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 -- 172.21.15.98:0/436556323 learned_addr learned my addr 172.21.15.98:0/436556323 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b3e00640 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42cf8c0 0x561cb42cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b3e00640 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42d78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 -- 172.21.15.98:0/436556323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42cf8c0 msgr2=0x561cb42cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42cf8c0 0x561cb42cfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 -- 172.21.15.98:0/436556323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 msgr2=0x561cb42d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42d78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 -- 172.21.15.98:0/436556323 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb41fe1f0 con 0x561cb42d02b0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.314+0000 7f96b8c00640 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009c90 tx=0x7f96a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d9f630b9fc432b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96a5c00640 1 -- 172.21.15.98:0/436556323 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8005220 con 0x561cb42d02b0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96a5c00640 1 -- 172.21.15.98:0/436556323 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8004c80 con 0x561cb42d02b0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96a5c00640 1 -- 172.21.15.98:0/436556323 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a80196d0 con 0x561cb42d02b0 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96b1600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 0x7f969c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96b1600640 1 -- 172.21.15.98:0/1783173242 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96ac0f1240 con 0x7f96b4109690 2024-03-16T12:23:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.315+0000 7f96b3e00640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 0x7f969c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- 172.21.15.98:0/436556323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 msgr2=0x561cb42d4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009c90 tx=0x7f96a8011810 comp rx=0 tx=0).stop 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- 172.21.15.98:0/436556323 shutdown_connections 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42d78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42d4db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 --2- 172.21.15.98:0/436556323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42cf8c0 0x561cb42cfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- 172.21.15.98:0/436556323 >> 172.21.15.98:0/436556323 conn(0x561cb42cae30 msgr2=0x561cb42cb230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- 172.21.15.98:0/436556323 shutdown_connections 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- 172.21.15.98:0/436556323 wait complete. 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96b3e00640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 0x7f969c0a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x561cb42d0b10 tx=0x7f96a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.316+0000 7f96be23e740 1 -- start start 2024-03-16T12:23:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 Processor -- start 2024-03-16T12:23:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 0x561cb42d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42e10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561cb41f0110 con 0x561cb42cf8c0 2024-03-16T12:23:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561cb42304f0 con 0x561cb42d5420 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561cb42d83d0 con 0x561cb42d02b0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96be23e740 10 client.0 _ll_register_callbacks cb 0x561cb41ddb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42e10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 0x561cb42d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55358/0 (socket says 172.21.15.98:55358) 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.317+0000 7f96b3e00640 1 -- 172.21.15.98:0/588034303 learned_addr learned my addr 172.21.15.98:0/588034303 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 -- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 msgr2=0x561cb42e10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42e10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 -- 172.21.15.98:0/588034303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 msgr2=0x561cb42d9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 0x561cb42d9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 -- 172.21.15.98:0/588034303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb3f483f0 con 0x561cb42d02b0 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b8c00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 0x561cb42d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b9600640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42e10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96b3e00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96a0010d10 tx=0x7f96a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc55cc31184dcd4f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f969a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a002e060 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96a0029600 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.318+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a8004c80 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f96a0019040 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 10 client.25960 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561cb40127d0 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8004c80 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f96a0028770 con 0x561cb42d02b0 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 10 client.25960 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 10 client.25960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 20 client.25960 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb42e5ae0 0x561cb42e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 4 client.25960 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.319+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb4241990 con 0x561cb42e5ae0 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.320+0000 7f96b8c00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb42e5ae0 0x561cb42e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.320+0000 7f969a000640 10 client.25960 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.320+0000 7f96b8c00640 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb42e5ae0 0x561cb42e7f60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89d79b24acb8defc server_cookie=47c8f875973d42b2 in_seq=0 out_seq=0 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f969a000640 1 -- 172.21.15.98:0/588034303 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cb4241990 con 0x561cb42e5ae0 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f969a000640 10 client.25960 handle_command_reply: tid=0 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 client.25960 shutdown 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 msgr2=0x561cb42da0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96a0010d10 tx=0x7f96a0004730 comp rx=0 tx=0).stop 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 shutdown_connections 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42e10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb42e5ae0 0x561cb42e7f60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42d02b0 0x561cb42da0a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f969a000640 0 client.25960 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 --2- 172.21.15.98:0/588034303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42cf8c0 0x561cb42d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 >> 172.21.15.98:0/588034303 conn(0x561cb42cae30 msgr2=0x561cb42ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 shutdown_connections 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.321+0000 7f96be23e740 1 -- 172.21.15.98:0/588034303 wait complete. 2024-03-16T12:23:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.322+0000 7f96be23e740 20 client.25960 trim_cache size 0 max 16384 2024-03-16T12:23:55.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:55 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/160214841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:23:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 Processor -- start 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 -- start start 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42ca070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42ec860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41aed60 con 0x561cb42d5420 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41ac630 con 0x561cb42d02b0 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.345+0000 7f96be23e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41aa480 con 0x561cb42cf8c0 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55640/0 (socket says 172.21.15.98:55640) 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 -- 172.21.15.98:0/3058002919 learned_addr learned my addr 172.21.15.98:0/3058002919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b9600640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 -- 172.21.15.98:0/3058002919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 msgr2=0x561cb42ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 -- 172.21.15.98:0/3058002919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 msgr2=0x561cb42ec860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42ec860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 -- 172.21.15.98:0/3058002919 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb42c6620 con 0x561cb42d02b0 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b8c00640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b9600640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42ec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96b3e00640 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009e80 tx=0x7f96a0009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a2f0780428a885 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.346+0000 7f96a5200640 1 -- 172.21.15.98:0/3058002919 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a002e050 con 0x561cb42d02b0 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.347+0000 7f96a5200640 1 -- 172.21.15.98:0/3058002919 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a001ec60 con 0x561cb42d02b0 2024-03-16T12:23:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.347+0000 7f96a5200640 1 -- 172.21.15.98:0/3058002919 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00047a0 con 0x561cb42d02b0 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- 172.21.15.98:0/3058002919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 msgr2=0x561cb42e10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009e80 tx=0x7f96a0009590 comp rx=0 tx=0).stop 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- 172.21.15.98:0/3058002919 shutdown_connections 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d5420 0x561cb42ec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d02b0 0x561cb42e10a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 --2- 172.21.15.98:0/3058002919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- 172.21.15.98:0/3058002919 >> 172.21.15.98:0/3058002919 conn(0x561cb42cae30 msgr2=0x561cb42c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- 172.21.15.98:0/3058002919 shutdown_connections 2024-03-16T12:23:55.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- 172.21.15.98:0/3058002919 wait complete. 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.348+0000 7f96be23e740 1 -- start start 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 Processor -- start 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d02b0 0x561cb42def20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41aaaf0 con 0x561cb42d02b0 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41abf60 con 0x561cb42d5420 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96b41ab1c0 con 0x561cb42cf8c0 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96be23e740 10 client.0 _ll_register_callbacks cb 0x7f96a0009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55646/0 (socket says 172.21.15.98:55646) 2024-03-16T12:23:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96b9600640 1 -- 172.21.15.98:0/705005739 learned_addr learned my addr 172.21.15.98:0/705005739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.349+0000 7f96b8c00640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55368/0 (socket says 172.21.15.98:55368) 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b3e00640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d02b0 0x561cb42def20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 -- 172.21.15.98:0/705005739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 msgr2=0x561cb42de9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 -- 172.21.15.98:0/705005739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d02b0 msgr2=0x561cb42def20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d02b0 0x561cb42def20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 -- 172.21.15.98:0/705005739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb42e0520 con 0x561cb42d5420 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b8c00640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96b9600640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003fd0 tx=0x7f96ac0bd360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=860c3ac0019b908b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f969a000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0098b0 con 0x561cb42d5420 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96ac0bb8c0 con 0x561cb42d5420 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a001ec60 con 0x561cb42d5420 2024-03-16T12:23:55.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f96ac0bd820 con 0x561cb42d5420 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96be23e740 10 client.25976 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.350+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561cb434c190 con 0x561cb42d5420 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a001ec60 con 0x561cb42d5420 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f96ac03c030 con 0x561cb42d5420 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 10 client.25976 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 10 client.25976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 20 client.25976 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb434cab0 0x561cb434eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 4 client.25976 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561cb423bce0 con 0x561cb434cab0 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.351+0000 7f96b8c00640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb434cab0 0x561cb434eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f969a000640 10 client.25976 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96b8c00640 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb434cab0 0x561cb434eed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e70be855e16ba26 server_cookie=43f523e6163c5eca in_seq=0 out_seq=0 2024-03-16T12:23:55.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f969a000640 1 -- 172.21.15.98:0/705005739 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561cb423bce0 con 0x561cb434cab0 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f969a000640 10 client.25976 handle_command_reply: tid=0 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 client.25976 shutdown 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 msgr2=0x561cb42dff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003fd0 tx=0x7f96ac0bd360 comp rx=0 tx=0).stop 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 shutdown_connections 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561cb42d5420 0x561cb42dff30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cb42d02b0 0x561cb42def20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x561cb434cab0 0x561cb434eed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 --2- 172.21.15.98:0/705005739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561cb42cf8c0 0x561cb42de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f969a000640 0 client.25976 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 >> 172.21.15.98:0/705005739 conn(0x561cb42cae30 msgr2=0x561cb42ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 shutdown_connections 2024-03-16T12:23:55.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 1 -- 172.21.15.98:0/705005739 wait complete. 2024-03-16T12:23:55.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.352+0000 7f96be23e740 20 client.25976 trim_cache size 0 max 16384 2024-03-16T12:23:55.338 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 msgr2=0x7f969c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 0x7f969c0a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x561cb42d0b10 tx=0x7f96a0005f90 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 msgr2=0x7f96b41a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b41a1260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96ac037040 tx=0x7f96ac0093d0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 shutdown_connections 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f969c0a2880 0x7f969c0a4d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4109690 0x7f96b41a1260 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96b40757a0 0x7f96b419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 --2- 172.21.15.98:0/1783173242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40741c0 0x7f96b4199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.353+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 >> 172.21.15.98:0/1783173242 conn(0x7f96b40fcd50 msgr2=0x7f96b4105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:55.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.354+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 shutdown_connections 2024-03-16T12:23:55.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:55.354+0000 7f96bb600640 1 -- 172.21.15.98:0/1783173242 wait complete. 2024-03-16T12:23:56.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:56 smithi153 ceph-mon[36901]: pgmap v477: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:56.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:56 smithi173 ceph-mon[36632]: pgmap v477: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:56.359 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T12:23:56.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:23:56.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T12:23:56.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:56 smithi098 ceph-mon[31633]: pgmap v477: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 MiB/s wr, 17 op/s 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 -- 172.21.15.98:0/4068031332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c8100a30 msgr2=0x7ff0c8100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 --2- 172.21.15.98:0/4068031332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c8100a30 0x7ff0c8100e90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009900 tx=0x7ff0b402bcd0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 -- 172.21.15.98:0/4068031332 shutdown_connections 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 --2- 172.21.15.98:0/4068031332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c81013d0 0x7ff0c8101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 --2- 172.21.15.98:0/4068031332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c8100a30 0x7ff0c8100e90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 --2- 172.21.15.98:0/4068031332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c81004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 -- 172.21.15.98:0/4068031332 >> 172.21.15.98:0/4068031332 conn(0x7ff0c80fbc70 msgr2=0x7ff0c80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 -- 172.21.15.98:0/4068031332 shutdown_connections 2024-03-16T12:23:56.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.598+0000 7ff0cf200640 1 -- 172.21.15.98:0/4068031332 wait complete. 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.599+0000 7ff0cf200640 1 Processor -- start 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.599+0000 7ff0cf200640 1 -- start start 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.599+0000 7ff0cf200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.599+0000 7ff0cf200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c8100a30 0x7ff0c819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0cf200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c81013d0 0x7ff0c81a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0cf200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810f690 con 0x7ff0c8100a30 2024-03-16T12:23:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0cf200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810f510 con 0x7ff0c81000d0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0cf200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c810f810 con 0x7ff0c81013d0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55662/0 (socket says 172.21.15.98:55662) 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c8100a30 0x7ff0c819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 -- 172.21.15.98:0/415858264 learned_addr learned my addr 172.21.15.98:0/415858264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0cd200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c81013d0 0x7ff0c81a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 -- 172.21.15.98:0/415858264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c81013d0 msgr2=0x7ff0c81a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c81013d0 0x7ff0c81a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 -- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c8100a30 msgr2=0x7ff0c819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c8100a30 0x7ff0c819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.600+0000 7ff0c7e00640 1 -- 172.21.15.98:0/415858264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c81a1930 con 0x7ff0c81000d0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0c7e00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800e9f0 tx=0x7ff0b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0c4c00640 1 -- 172.21.15.98:0/415858264 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b800cd80 con 0x7ff0c81000d0 2024-03-16T12:23:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0c4c00640 1 -- 172.21.15.98:0/415858264 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b8010910 con 0x7ff0c81000d0 2024-03-16T12:23:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c81a1c20 con 0x7ff0c81000d0 2024-03-16T12:23:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0c4c00640 1 -- 172.21.15.98:0/415858264 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80188f0 con 0x7ff0c81000d0 2024-03-16T12:23:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.601+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81a21c0 con 0x7ff0c81000d0 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 Processor -- start 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 -- start start 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0c4c00640 1 -- 172.21.15.98:0/415858264 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff0b8010ac0 con 0x7ff0c81000d0 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e30fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e310340 0x563f2e3127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563f2e26df80 con 0x563f2e310340 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563f2e216ad0 con 0x563f2e30b1d0 2024-03-16T12:23:56.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563f2e26a020 con 0x563f2e30a7e0 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0c7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e30fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0cd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.605+0000 7ff0cd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55386/0 (socket says 172.21.15.98:55386) 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 -- 172.21.15.98:0/1223275956 learned_addr learned my addr 172.21.15.98:0/1223275956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0c7400640 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e310340 0x563f2e3127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 -- 172.21.15.98:0/1223275956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 msgr2=0x563f2e30fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e30fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 -- 172.21.15.98:0/1223275956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e310340 msgr2=0x563f2e3127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e310340 0x563f2e3127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 -- 172.21.15.98:0/1223275956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f2e22e6f0 con 0x563f2e30a7e0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0cd200640 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009fd0 tx=0x7ff0c00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fffce7ee69e7940 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0bd600640 1 -- 172.21.15.98:0/1223275956 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0004d20 con 0x563f2e30a7e0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0bd600640 1 -- 172.21.15.98:0/1223275956 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0c0004050 con 0x563f2e30a7e0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.606+0000 7ff0bd600640 1 -- 172.21.15.98:0/1223275956 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0005260 con 0x563f2e30a7e0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0c4c00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 0x7ff0ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0c4c00640 1 -- 172.21.15.98:0/415858264 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0b80cf1b0 con 0x7ff0c81000d0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0c7e00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 0x7ff0ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 -- 172.21.15.98:0/1223275956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 msgr2=0x563f2e30ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009fd0 tx=0x7ff0c00118d0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 -- 172.21.15.98:0/1223275956 shutdown_connections 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e310340 0x563f2e3127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e30fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/1223275956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e30ac00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.607+0000 7ff0d1e65740 1 -- 172.21.15.98:0/1223275956 >> 172.21.15.98:0/1223275956 conn(0x563f2e2d2930 msgr2=0x563f2e306150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.608+0000 7ff0d1e65740 1 -- 172.21.15.98:0/1223275956 shutdown_connections 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.608+0000 7ff0d1e65740 1 -- 172.21.15.98:0/1223275956 wait complete. 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.608+0000 7ff0c7e00640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 0x7ff0ac0a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff0b809b540 tx=0x7ff0b80d2bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.608+0000 7ff0d1e65740 1 -- start start 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.608+0000 7ff0d1e65740 1 Processor -- start 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30b1d0 0x563f2e315020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e310340 0x563f2e31c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563f2e3132f0 con 0x563f2e30b1d0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563f2e229f30 con 0x563f2e310340 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563f2e313470 con 0x563f2e30a7e0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0d1e65740 10 client.0 _ll_register_callbacks cb 0x563f2e217bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55394/0 (socket says 172.21.15.98:55394) 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 -- 172.21.15.98:0/4169295720 learned_addr learned my addr 172.21.15.98:0/4169295720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e310340 0x563f2e31c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 -- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e310340 msgr2=0x563f2e31c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0c7400640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30b1d0 0x563f2e315020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e310340 0x563f2e31c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 -- 172.21.15.98:0/4169295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30b1d0 msgr2=0x563f2e315020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30b1d0 0x563f2e315020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.609+0000 7ff0cd200640 1 -- 172.21.15.98:0/4169295720 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f2e04a7e0 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0cd200640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009de0 tx=0x7ff0c0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ad27339d1ce150f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c000f4e0 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0c000f680 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0c0004050 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff0c0036050 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0d1e65740 10 client.25978 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563f2e0aaf70 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0c0004050 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.610+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff0c0025040 con 0x563f2e30a7e0 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 10 client.25978 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 10 client.25978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 20 client.25978 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e320720 0x563f2e322b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 4 client.25978 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f2e27b4c0 con 0x563f2e320720 2024-03-16T12:23:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0c7400640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e320720 0x563f2e322b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0c7400640 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e320720 0x563f2e322b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd8b3b28c9888897 server_cookie=6d1caed137c13394 in_seq=0 out_seq=0 2024-03-16T12:23:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.611+0000 7ff0b2000640 10 client.25978 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.612+0000 7ff0b2000640 1 -- 172.21.15.98:0/4169295720 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f2e27b4c0 con 0x563f2e320720 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.612+0000 7ff0b2000640 10 client.25978 handle_command_reply: tid=0 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.612+0000 7ff0d1e65740 1 client.25978 shutdown 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 msgr2=0x563f2e314ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009de0 tx=0x7ff0c0000f40 comp rx=0 tx=0).stop 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 shutdown_connections 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e310340 0x563f2e31c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30b1d0 0x563f2e315020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e320720 0x563f2e322b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/4169295720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e30a7e0 0x563f2e314ae0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0b2000640 0 client.25978 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 >> 172.21.15.98:0/4169295720 conn(0x563f2e2d2930 msgr2=0x563f2e305350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 shutdown_connections 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 1 -- 172.21.15.98:0/4169295720 wait complete. 2024-03-16T12:23:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.613+0000 7ff0d1e65740 20 client.25978 trim_cache size 0 max 16384 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 Processor -- start 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 -- start start 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e314130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e314e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81aee00 con 0x563f2e30a7e0 2024-03-16T12:23:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81ac6d0 con 0x563f2e30b1d0 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.637+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81aa520 con 0x563f2e310340 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55688/0 (socket says 172.21.15.98:55688) 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 -- 172.21.15.98:0/3800421525 learned_addr learned my addr 172.21.15.98:0/3800421525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e314130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e314e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 -- 172.21.15.98:0/3800421525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 msgr2=0x563f2e314e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e314e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 -- 172.21.15.98:0/3800421525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 msgr2=0x563f2e314130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e314130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 -- 172.21.15.98:0/3800421525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f2df42170 con 0x563f2e30b1d0 2024-03-16T12:23:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e314130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:23:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0b402c850 tx=0x7ff0b402c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ba145a384835 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.638+0000 7ff0bcc00640 1 -- 172.21.15.98:0/3800421525 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b404b070 con 0x563f2e30b1d0 2024-03-16T12:23:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0bcc00640 1 -- 172.21.15.98:0/3800421525 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b4004030 con 0x563f2e30b1d0 2024-03-16T12:23:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0bcc00640 1 -- 172.21.15.98:0/3800421525 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4004f10 con 0x563f2e30b1d0 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3800421525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 msgr2=0x563f2e31c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0b402c850 tx=0x7ff0b402c920 comp rx=0 tx=0).stop 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3800421525 shutdown_connections 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e314e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e31c020 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3800421525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e314130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3800421525 >> 172.21.15.98:0/3800421525 conn(0x563f2e2d2930 msgr2=0x563f2e3146f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.639+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3800421525 shutdown_connections 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3800421525 wait complete. 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 -- start start 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 Processor -- start 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e3195c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e31ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81aab90 con 0x563f2e30a7e0 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81ac000 con 0x563f2e30b1d0 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81ab260 con 0x563f2e310340 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.640+0000 7ff0d1e65740 10 client.0 _ll_register_callbacks cb 0x7ff0b80d2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0cd200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e3195c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:55704/0 (socket says 172.21.15.98:55704) 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 -- 172.21.15.98:0/3832999269 learned_addr learned my addr 172.21.15.98:0/3832999269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e31ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 -- 172.21.15.98:0/3832999269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 msgr2=0x563f2e31ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e31ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 -- 172.21.15.98:0/3832999269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 msgr2=0x563f2e3195c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e3195c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 -- 172.21.15.98:0/3832999269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f2e31b100 con 0x563f2e30b1d0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0c7400640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009230 tx=0x7ff0b4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea7440114266fa8a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.641+0000 7ff0b2000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4054060 con 0x563f2e30b1d0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0b4004030 con 0x563f2e30b1d0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0b40043e0 con 0x563f2e30b1d0 2024-03-16T12:23:56.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff0b404b050 con 0x563f2e30b1d0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 10 client.25994 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563f2e386a90 con 0x563f2e30b1d0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b4004030 con 0x563f2e30b1d0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff0b40035f0 con 0x563f2e30b1d0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 10 client.25994 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 10 client.25994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 20 client.25994 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e3872d0 0x563f2e3896f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 4 client.25994 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.642+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563f2e275810 con 0x563f2e3872d0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.643+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e3872d0 0x563f2e3896f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.643+0000 7ff0cd200640 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e3872d0 0x563f2e3896f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d63c0933a6136fa0 server_cookie=9e68e0bb755356bb in_seq=0 out_seq=0 2024-03-16T12:23:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.643+0000 7ff0b2000640 10 client.25994 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0b2000640 1 -- 172.21.15.98:0/3832999269 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563f2e275810 con 0x563f2e3872d0 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0b2000640 10 client.25994 handle_command_reply: tid=0 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 client.25994 shutdown 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 msgr2=0x563f2e319b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009230 tx=0x7ff0b4005fd0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 shutdown_connections 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x563f2e3872d0 0x563f2e3896f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f2e310340 0x563f2e31ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563f2e30b1d0 0x563f2e319b00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 --2- 172.21.15.98:0/3832999269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563f2e30a7e0 0x563f2e3195c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 >> 172.21.15.98:0/3832999269 conn(0x563f2e2d2930 msgr2=0x563f2e304ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.644+0000 7ff0b2000640 0 client.25994 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 shutdown_connections 2024-03-16T12:23:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0d1e65740 1 -- 172.21.15.98:0/3832999269 wait complete. 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0d1e65740 20 client.25994 trim_cache size 0 max 16384 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "4e858ed9-2517-4708-9980-e68e9e31b9b0", 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-03-16T12:23:56.631 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 msgr2=0x7ff0ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 0x7ff0ac0a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff0b809b540 tx=0x7ff0b80d2bf0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 msgr2=0x7ff0c8199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800e9f0 tx=0x7ff0b800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 shutdown_connections 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff0ac0a2880 0x7ff0ac0a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c81013d0 0x7ff0c81a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0c8100a30 0x7ff0c819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 --2- 172.21.15.98:0/415858264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0c81000d0 0x7ff0c8199d20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.645+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 >> 172.21.15.98:0/415858264 conn(0x7ff0c80fbc70 msgr2=0x7ff0c80fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.646+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 shutdown_connections 2024-03-16T12:23:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.646+0000 7ff0cf200640 1 -- 172.21.15.98:0/415858264 wait complete. 2024-03-16T12:23:56.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:23:56.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:23:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 -- 172.21.15.98:0/2620491537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 msgr2=0x7f8a44101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2620491537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44101c10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009900 tx=0x7f8a3802bc50 comp rx=0 tx=0).stop 2024-03-16T12:23:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 -- 172.21.15.98:0/2620491537 shutdown_connections 2024-03-16T12:23:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2620491537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441033a0 0x7f8a44109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2620491537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441029e0 0x7f8a44102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2620491537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44101c10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 -- 172.21.15.98:0/2620491537 >> 172.21.15.98:0/2620491537 conn(0x7f8a440fcd80 msgr2=0x7f8a440ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 -- 172.21.15.98:0/2620491537 shutdown_connections 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.866+0000 7f8a4b800640 1 -- 172.21.15.98:0/2620491537 wait complete. 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.867+0000 7f8a4b800640 1 Processor -- start 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 -- start start 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441029e0 0x7f8a4419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441033a0 0x7f8a441a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4410d490 con 0x7f8a441033a0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4410d310 con 0x7f8a441017f0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a4b800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4410d610 con 0x7f8a441029e0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50610/0 (socket says 172.21.15.98:50610) 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a48e00640 1 -- 172.21.15.98:0/2317994406 learned_addr learned my addr 172.21.15.98:0/2317994406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.868+0000 7f8a43e00640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441029e0 0x7f8a4419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a49800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441033a0 0x7f8a441a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 -- 172.21.15.98:0/2317994406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441029e0 msgr2=0x7f8a4419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441029e0 0x7f8a4419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 -- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441033a0 msgr2=0x7f8a441a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441033a0 0x7f8a441a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 -- 172.21.15.98:0/2317994406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a441a1960 con 0x7f8a441017f0 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005ab0 tx=0x7f8a3802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a41600640 1 -- 172.21.15.98:0/2317994406 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38041070 con 0x7f8a441017f0 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a41600640 1 -- 172.21.15.98:0/2317994406 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a38004450 con 0x7f8a441017f0 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a41600640 1 -- 172.21.15.98:0/2317994406 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38034440 con 0x7f8a441017f0 2024-03-16T12:23:56.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a441a1bf0 con 0x7f8a441017f0 2024-03-16T12:23:56.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.869+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a441a20f0 con 0x7f8a441017f0 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.873+0000 7f8a41600640 1 -- 172.21.15.98:0/2317994406 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8a38044070 con 0x7f8a441017f0 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 Processor -- start 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a41600640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 0x7f8a2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 -- start start 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a41600640 1 -- 172.21.15.98:0/2317994406 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8a380f09b0 con 0x7f8a441017f0 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a49800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 0x7f8a2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f905fdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905fe370 0x564f90600800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f9055d1f0 con 0x564f905fe370 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f90505a80 con 0x564f905f9200 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.874+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f90559290 con 0x564f905f8810 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56780/0 (socket says 172.21.15.98:56780) 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 -- 172.21.15.98:0/1391182642 learned_addr learned my addr 172.21.15.98:0/1391182642 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a49800640 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905fe370 0x564f90600800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a43e00640 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f905fdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 -- 172.21.15.98:0/1391182642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 msgr2=0x564f905fdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f905fdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 -- 172.21.15.98:0/1391182642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905fe370 msgr2=0x564f90600800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905fe370 0x564f90600800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 -- 172.21.15.98:0/1391182642 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f9033b7d0 con 0x564f905f8810 2024-03-16T12:23:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a48e00640 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a380bcf20 tx=0x7f8a38003a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3d798bbcf7d2002 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.875+0000 7f8a49800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 0x7f8a2c0a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005fd0 tx=0x7f8a3c013a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a36000640 1 -- 172.21.15.98:0/1391182642 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3804b040 con 0x564f905f8810 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a36000640 1 -- 172.21.15.98:0/1391182642 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a36000640 1 -- 172.21.15.98:0/1391182642 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380c52a0 con 0x564f905f8810 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/1391182642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 msgr2=0x564f905f8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a380bcf20 tx=0x7f8a38003a20 comp rx=0 tx=0).stop 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/1391182642 shutdown_connections 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905fe370 0x564f90600800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f905fdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/1391182642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f905f8c30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.876+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/1391182642 >> 172.21.15.98:0/1391182642 conn(0x564f905f3d80 msgr2=0x564f905f4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.877+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/1391182642 shutdown_connections 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.877+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/1391182642 wait complete. 2024-03-16T12:23:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.877+0000 7f8a4e5ee740 1 -- start start 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.877+0000 7f8a4e5ee740 1 Processor -- start 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90603010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f9060a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564f906010e0 con 0x564f905f9200 2024-03-16T12:23:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564f90518ed0 con 0x564f905f8810 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564f90601260 con 0x564f905fe370 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a4e5ee740 10 client.0 _ll_register_callbacks cb 0x564f90554220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90603010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a49800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f9060a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50616/0 (socket says 172.21.15.98:50616) 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.878+0000 7f8a48e00640 1 -- 172.21.15.98:0/3570236575 learned_addr learned my addr 172.21.15.98:0/3570236575 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 -- 172.21.15.98:0/3570236575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 msgr2=0x564f9060a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f9060a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 -- 172.21.15.98:0/3570236575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 msgr2=0x564f90603010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90603010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 -- 172.21.15.98:0/3570236575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f902713f0 con 0x564f905f8810 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a48e00640 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a380096c0 tx=0x7f8a380bd110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5463a0e6e02414ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3804b050 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8a380c54a0 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8a380bc3e0 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a4e5ee740 10 client.26006 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f902310f0 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.879+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8a380bbe50 con 0x564f905f8810 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 10 client.26006 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 20 client.26006 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f9060e540 0x564f90610930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 4 client.26006 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:23:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f90570be0 con 0x564f9060e540 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a43e00640 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f9060e540 0x564f90610930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a43e00640 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f9060e540 0x564f90610930 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ecd8f4457bd419d server_cookie=3e21ce50425997cd in_seq=0 out_seq=0 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.880+0000 7f8a2aa00640 10 client.26006 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a2aa00640 1 -- 172.21.15.98:0/3570236575 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f90570be0 con 0x564f9060e540 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a2aa00640 10 client.26006 handle_command_reply: tid=0 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 client.26006 shutdown 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 msgr2=0x564f90602ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a380096c0 tx=0x7f8a380bd110 comp rx=0 tx=0).stop 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 shutdown_connections 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f9060e540 0x564f90610930 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f9060a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90603010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a2aa00640 0 client.26006 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/3570236575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f8810 0x564f90602ad0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 >> 172.21.15.98:0/3570236575 conn(0x564f905f3d80 msgr2=0x564f905f2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.881+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 shutdown_connections 2024-03-16T12:23:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.882+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/3570236575 wait complete. 2024-03-16T12:23:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.882+0000 7f8a4e5ee740 20 client.26006 trim_cache size 0 max 16384 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.900+0000 7f8a4e5ee740 1 Processor -- start 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.900+0000 7f8a4e5ee740 1 -- start start 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90609fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 0x564f905f2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441aeea0 con 0x564f905f9200 2024-03-16T12:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441ac770 con 0x564f905fe370 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441aa5c0 con 0x564f905f8810 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90609fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a49800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 0x564f905f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56798/0 (socket says 172.21.15.98:56798) 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 -- 172.21.15.98:0/2102657811 learned_addr learned my addr 172.21.15.98:0/2102657811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 -- 172.21.15.98:0/2102657811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 msgr2=0x564f905f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 0x564f905f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 -- 172.21.15.98:0/2102657811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 msgr2=0x564f90609fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90609fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a48e00640 1 -- 172.21.15.98:0/2102657811 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f905f4700 con 0x564f905f8810 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.901+0000 7f8a49800640 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 0x564f905f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.902+0000 7f8a48e00640 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a380b9960 tx=0x7f8a38003a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8cc701d487aefe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.902+0000 7f8a35600640 1 -- 172.21.15.98:0/2102657811 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3804b070 con 0x564f905f8810 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.902+0000 7f8a35600640 1 -- 172.21.15.98:0/2102657811 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.902+0000 7f8a35600640 1 -- 172.21.15.98:0/2102657811 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380bbe50 con 0x564f905f8810 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/2102657811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 msgr2=0x564f90602df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a380b9960 tx=0x7f8a38003a20 comp rx=0 tx=0).stop 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/2102657811 shutdown_connections 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905fe370 0x564f905f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f9200 0x564f90609fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/2102657811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905f8810 0x564f90602df0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/2102657811 >> 172.21.15.98:0/2102657811 conn(0x564f905f3d80 msgr2=0x564f906026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/2102657811 shutdown_connections 2024-03-16T12:23:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/2102657811 wait complete. 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 -- start start 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.903+0000 7f8a4e5ee740 1 Processor -- start 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f90608540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441aac30 con 0x564f905f8810 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441ac0a0 con 0x564f905f9200 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441ab300 con 0x564f905fe370 2024-03-16T12:23:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a4e5ee740 10 client.0 _ll_register_callbacks cb 0x7f8a3000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a49800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f90608540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50424/0 (socket says 172.21.15.98:50424) 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:50632/0 (socket says 172.21.15.98:50632) 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 -- 172.21.15.98:0/982089434 learned_addr learned my addr 172.21.15.98:0/982089434 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 -- 172.21.15.98:0/982089434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 msgr2=0x564f90608540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f90608540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 -- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 msgr2=0x564f90607580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a48e00640 1 -- 172.21.15.98:0/982089434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f90608b30 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.904+0000 7f8a43e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a48e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8a3802c190 tx=0x7f8a380bd110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7c509d83313c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3804b030 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8a380c52a0 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8a380bc8f0 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a4e5ee740 10 client.15840 fetch_fsmap learned FSMap version 24 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f90608b30 con 0x564f905f8810 2024-03-16T12:23:56.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.905+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a380c5730 con 0x564f905f8810 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8a380b8ce0 con 0x564f905f8810 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 10 client.15840 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 20 client.15840 populate_metadata read hostname 'smithi098' 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f906747b0 0x564f90676ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 4 client.15840 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f9056a730 con 0x564f906747b0 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a43e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f906747b0 0x564f90676ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a2aa00640 10 client.15840 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.906+0000 7f8a43e00640 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f906747b0 0x564f90676ba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9ca6d53aa780ffd server_cookie=a07b12135dd5a44b in_seq=0 out_seq=0 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a2aa00640 1 -- 172.21.15.98:0/982089434 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f9056a730 con 0x564f906747b0 2024-03-16T12:23:56.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a2aa00640 10 client.15840 handle_command_reply: tid=0 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 client.15840 shutdown 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 msgr2=0x564f90607040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8a3802c190 tx=0x7f8a380bd110 comp rx=0 tx=0).stop 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 shutdown_connections 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x564f906747b0 0x564f90676ba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564f905fe370 0x564f90608540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564f905f9200 0x564f90607580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a2aa00640 0 client.15840 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 --2- 172.21.15.98:0/982089434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564f905f8810 0x564f90607040 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 >> 172.21.15.98:0/982089434 conn(0x564f905f3d80 msgr2=0x564f905f6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 shutdown_connections 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 1 -- 172.21.15.98:0/982089434 wait complete. 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.907+0000 7f8a4e5ee740 20 client.15840 trim_cache size 0 max 16384 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stdout: "4e858ed9-2517-4708-9980-e68e9e31b9b0": { 2024-03-16T12:23:56.896 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-03-16T12:23:56.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "4e858ed9-2517-4708-9980-e68e9e31b9b0", 2024-03-16T12:23:56.897 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-03-16T12:23:56.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:23:56.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:23:56.897 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 msgr2=0x7f8a2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 0x7f8a2c0a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005fd0 tx=0x7f8a3c013a40 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 msgr2=0x7f8a44199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005ab0 tx=0x7f8a3802c6e0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 shutdown_connections 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f8a2c0a2880 0x7f8a2c0a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441033a0 0x7f8a441a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a441029e0 0x7f8a4419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 --2- 172.21.15.98:0/2317994406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a441017f0 0x7f8a44199d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:23:56.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 >> 172.21.15.98:0/2317994406 conn(0x7f8a440fcd80 msgr2=0x7f8a44073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:23:56.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 shutdown_connections 2024-03-16T12:23:56.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:23:56.908+0000 7f8a4b800640 1 -- 172.21.15.98:0/2317994406 wait complete. 2024-03-16T12:23:56.906 INFO:tasks.cephfs.filesystem:scrub status for tag:4e858ed9-2517-4708-9980-e68e9e31b9b0 - {'path': '/', 'tag': '4e858ed9-2517-4708-9980-e68e9e31b9b0', 'options': 'recursive,force'} 2024-03-16T12:23:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:58 smithi173 ceph-mon[36632]: scrub queued for path: / 2024-03-16T12:23:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:58 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:58 smithi173 ceph-mon[36632]: scrub summary: active paths [/] 2024-03-16T12:23:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:58 smithi173 ceph-mon[36632]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:23:58 smithi173 ceph-mon[36632]: pgmap v478: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 15 op/s 2024-03-16T12:23:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:58 smithi098 ceph-mon[31633]: scrub queued for path: / 2024-03-16T12:23:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:58 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:58 smithi098 ceph-mon[31633]: scrub summary: active paths [/] 2024-03-16T12:23:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:58 smithi098 ceph-mon[31633]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:23:58 smithi098 ceph-mon[31633]: pgmap v478: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 15 op/s 2024-03-16T12:23:58.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:58 smithi153 ceph-mon[36901]: scrub queued for path: / 2024-03-16T12:23:58.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:58 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:58 smithi153 ceph-mon[36901]: scrub summary: active paths [/] 2024-03-16T12:23:58.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:58 smithi153 ceph-mon[36901]: scrub summary: idle+waiting paths [/] 2024-03-16T12:23:58.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:23:58 smithi153 ceph-mon[36901]: pgmap v478: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 15 op/s 2024-03-16T12:24:00.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:00 smithi173 ceph-mon[36632]: pgmap v479: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:00.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:00 smithi173 ceph-mon[36632]: scrub summary: idle 2024-03-16T12:24:00.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:00 smithi098 ceph-mon[31633]: pgmap v479: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:00.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:00 smithi098 ceph-mon[31633]: scrub summary: idle 2024-03-16T12:24:00.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:00 smithi153 ceph-mon[36901]: pgmap v479: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:00.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:00 smithi153 ceph-mon[36901]: scrub summary: idle 2024-03-16T12:24:02.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:02 smithi173 ceph-mon[36632]: pgmap v480: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:24:02.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:02 smithi098 ceph-mon[31633]: pgmap v480: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:24:02.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:02 smithi153 ceph-mon[36901]: pgmap v480: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-03-16T12:24:04.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:04 smithi173 ceph-mon[36632]: pgmap v481: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 10 op/s 2024-03-16T12:24:04.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:04 smithi098 ceph-mon[31633]: pgmap v481: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 10 op/s 2024-03-16T12:24:04.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:04 smithi153 ceph-mon[36901]: pgmap v481: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 862 KiB/s wr, 10 op/s 2024-03-16T12:24:06.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:06 smithi153 ceph-mon[36901]: pgmap v482: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:24:06.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:06 smithi173 ceph-mon[36632]: pgmap v482: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:24:06.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:06 smithi098 ceph-mon[31633]: pgmap v482: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 859 KiB/s wr, 9 op/s 2024-03-16T12:24:08.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:08 smithi173 ceph-mon[36632]: pgmap v483: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:24:08.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:08 smithi098 ceph-mon[31633]: pgmap v483: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:24:08.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:08 smithi153 ceph-mon[36901]: pgmap v483: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-03-16T12:24:10.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:10 smithi173 ceph-mon[36632]: pgmap v484: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:10.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:10 smithi098 ceph-mon[31633]: pgmap v484: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:10.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:10 smithi153 ceph-mon[36901]: pgmap v484: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 859 KiB/s wr, 8 op/s 2024-03-16T12:24:11.936 INFO:tasks.workunit.client.0.smithi098.stdout:setting up random write file 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:done setting up random write file 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:starting fsync run 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0010s fsync time: 0.0346s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0009s fsync time: 0.0244s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0263s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0255s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0226s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0202s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0199s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0197s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0209s 2024-03-16T12:24:11.937 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0006s fsync time: 0.0109s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0009s fsync time: 0.0214s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0190s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0209s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0222s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0221s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0202s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0006s fsync time: 0.0108s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0241s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0019s fsync time: 0.0279s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0219s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0229s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0210s 2024-03-16T12:24:11.938 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0277s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0195s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0191s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0219s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0195s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0190s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0213s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0177s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0162s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0205s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0209s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0199s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0202s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0207s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0192s 2024-03-16T12:24:11.939 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0220s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0192s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0270s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0161s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0254s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0007s fsync time: 0.0130s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0210s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0009s fsync time: 0.0114s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0223s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0269s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0221s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0227s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0209s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0199s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0171s 2024-03-16T12:24:11.940 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0200s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0198s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0008s fsync time: 0.0174s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0012s fsync time: 0.0215s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0199s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0207s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0013s fsync time: 0.0191s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:write time: 0.0014s fsync time: 0.0202s 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stdout:run done 60 fsyncs total, killing random writer 2024-03-16T12:24:11.941 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-16T12:24:11.942 INFO:tasks.workunit.client.0.smithi098.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-16T12:24:11.942 INFO:tasks.workunit.client.0.smithi098.stderr:+ whereis lsof 2024-03-16T12:24:11.987 INFO:tasks.workunit.client.0.smithi098.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-03-16T12:24:11.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ lsof 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/1de6dd084341ae09402ac108ef31c8894d2f6a27b316ccd7947fb6e29fc6b17a/merged 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/cd1201717d99b49614b0ab4c6ff0d602a9c63a6caba92bab64d2a42542dd69aa/merged 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/5e39474d4091204c702c8a48b0c88d17ba22862db03148ba95f8c3bf0768ecf8/merged 2024-03-16T12:24:12.016 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/351a1bffdf5463228fa7297813357e4e24fa189591e27b53ea87f6b5aeac1736/merged 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/3fdcd4848093ec27b18a2e2444d2594b921a73f720b9bf6649b299e8f12887a9/merged 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/23ba5fb53ed0b4fbd315da8b238e2949be5eefad04641e090eae44dc56764330/merged 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/44de55244f189f83a7a4c2b0e1650e4e5103e33006f2f10a9a9bc2dfa97ff7d5/merged 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/849d0b4287b5ce839b784cc617e9f95cecafbf19ab903d6ca8b695e916156819/merged 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.023 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/df36a0ce355b793307696f174c14f47d9f3c17b52ead629a271ebcb7c0d1f666/merged 2024-03-16T12:24:12.024 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.024 INFO:tasks.workunit.client.0.smithi098.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/a7ba7be87cbf090895649102087388e66cd280145dcf7254381e23457f821af8/merged 2024-03-16T12:24:12.024 INFO:tasks.workunit.client.0.smithi098.stderr: Output information may be incomplete. 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-03-16T12:24:12.155 INFO:tasks.workunit.client.0.smithi098.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:pool_work 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:pool_work 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:pool_work 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:pool_work 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-03-16T12:24:12.156 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 7 root cwd unknown /proc/7/cwd (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 7 root rtd unknown /proc/7/root (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 7 root txt unknown /proc/7/exe (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 7 root NOFD /proc/7/fd (opendir: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-03-16T12:24:12.157 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 11 root cwd unknown /proc/11/cwd (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 11 root rtd unknown /proc/11/root (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 11 root txt unknown /proc/11/exe (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 11 root NOFD /proc/11/fd (opendir: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-03-16T12:24:12.158 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_tasks 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-03-16T12:24:12.159 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/0 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/0 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/0 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/0 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/1 20 root cwd unknown /proc/20/cwd (readlink: Permission denied) 2024-03-16T12:24:12.160 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/1 20 root rtd unknown /proc/20/root (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/1 20 root txt unknown /proc/20/exe (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/1 20 root NOFD /proc/20/fd (opendir: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:migration 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:migration 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:migration 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:migration 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 23 root cwd unknown /proc/23/cwd (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 23 root rtd unknown /proc/23/root (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 23 root txt unknown /proc/23/exe (readlink: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 23 root NOFD /proc/23/fd (opendir: Permission denied) 2024-03-16T12:24:12.161 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/2 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/2 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/2 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/2 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:migration 26 root cwd unknown /proc/26/cwd (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:migration 26 root rtd unknown /proc/26/root (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:migration 26 root txt unknown /proc/26/exe (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:migration 26 root NOFD /proc/26/fd (opendir: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-03-16T12:24:12.162 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/3 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/3 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/3 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/3 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:migration 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:migration 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:migration 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:migration 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-03-16T12:24:12.163 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 32 root cwd unknown /proc/32/cwd (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 32 root rtd unknown /proc/32/root (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 32 root txt unknown /proc/32/exe (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 32 root NOFD /proc/32/fd (opendir: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/4 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/4 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/4 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/4 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-03-16T12:24:12.164 INFO:tasks.workunit.client.0.smithi098.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/5 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/5 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/5 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/5 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-03-16T12:24:12.165 INFO:tasks.workunit.client.0.smithi098.stdout:migration 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:migration 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:migration 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:migration 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 44 root cwd unknown /proc/44/cwd (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 44 root rtd unknown /proc/44/root (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 44 root txt unknown /proc/44/exe (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 44 root NOFD /proc/44/fd (opendir: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/6 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/6 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/6 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-03-16T12:24:12.166 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/6 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:migration 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:migration 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:migration 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:migration 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-03-16T12:24:12.167 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/7 50 root cwd unknown /proc/50/cwd (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/7 50 root rtd unknown /proc/50/root (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/7 50 root txt unknown /proc/50/exe (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:cpuhp/7 50 root NOFD /proc/50/fd (opendir: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:migration 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:migration 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:migration 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:migration 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:ksoftirqd 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-03-16T12:24:12.168 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kdevtmpfs 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kdevtmpfs 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kdevtmpfs 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kdevtmpfs 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 56 root cwd unknown /proc/56/cwd (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 56 root rtd unknown /proc/56/root (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 56 root txt unknown /proc/56/exe (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 56 root NOFD /proc/56/fd (opendir: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kauditd 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kauditd 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kauditd 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:kauditd 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-03-16T12:24:12.169 INFO:tasks.workunit.client.0.smithi098.stdout:khungtask 59 root cwd unknown /proc/59/cwd (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:khungtask 59 root rtd unknown /proc/59/root (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:khungtask 59 root txt unknown /proc/59/exe (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:khungtask 59 root NOFD /proc/59/fd (opendir: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:oom_reape 61 root cwd unknown /proc/61/cwd (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:oom_reape 61 root rtd unknown /proc/61/root (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:oom_reape 61 root txt unknown /proc/61/exe (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:oom_reape 61 root NOFD /proc/61/fd (opendir: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kcompactd 64 root cwd unknown /proc/64/cwd (readlink: Permission denied) 2024-03-16T12:24:12.170 INFO:tasks.workunit.client.0.smithi098.stdout:kcompactd 64 root rtd unknown /proc/64/root (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kcompactd 64 root txt unknown /proc/64/exe (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kcompactd 64 root NOFD /proc/64/fd (opendir: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:ksmd 65 root cwd unknown /proc/65/cwd (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:ksmd 65 root rtd unknown /proc/65/root (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:ksmd 65 root txt unknown /proc/65/exe (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:ksmd 65 root NOFD /proc/65/fd (opendir: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:khugepage 66 root cwd unknown /proc/66/cwd (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:khugepage 66 root rtd unknown /proc/66/root (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:khugepage 66 root txt unknown /proc/66/exe (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:khugepage 66 root NOFD /proc/66/fd (opendir: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-03-16T12:24:12.171 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 68 root cwd unknown /proc/68/cwd (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 68 root rtd unknown /proc/68/root (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 68 root txt unknown /proc/68/exe (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 68 root NOFD /proc/68/fd (opendir: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 69 root cwd unknown /proc/69/cwd (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 69 root rtd unknown /proc/69/root (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 69 root txt unknown /proc/69/exe (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 69 root NOFD /proc/69/fd (opendir: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 70 root cwd unknown /proc/70/cwd (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 70 root rtd unknown /proc/70/root (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 70 root txt unknown /proc/70/exe (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 70 root NOFD /proc/70/fd (opendir: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 72 root cwd unknown /proc/72/cwd (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 72 root rtd unknown /proc/72/root (readlink: Permission denied) 2024-03-16T12:24:12.172 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 72 root txt unknown /proc/72/exe (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 72 root NOFD /proc/72/fd (opendir: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 75 root cwd unknown /proc/75/cwd (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 75 root rtd unknown /proc/75/root (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 75 root txt unknown /proc/75/exe (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 75 root NOFD /proc/75/fd (opendir: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:irq/9-acp 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:irq/9-acp 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:irq/9-acp 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:irq/9-acp 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 79 root cwd unknown /proc/79/cwd (readlink: Permission denied) 2024-03-16T12:24:12.173 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 79 root rtd unknown /proc/79/root (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 79 root txt unknown /proc/79/exe (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 79 root NOFD /proc/79/fd (opendir: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-03-16T12:24:12.174 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:watchdogd 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:watchdogd 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:watchdogd 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:watchdogd 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 85 root cwd unknown /proc/85/cwd (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 85 root rtd unknown /proc/85/root (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 85 root txt unknown /proc/85/exe (readlink: Permission denied) 2024-03-16T12:24:12.175 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 85 root NOFD /proc/85/fd (opendir: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kswapd0 86 root cwd unknown /proc/86/cwd (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kswapd0 86 root rtd unknown /proc/86/root (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kswapd0 86 root txt unknown /proc/86/exe (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kswapd0 86 root NOFD /proc/86/fd (opendir: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 88 root cwd unknown /proc/88/cwd (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 88 root rtd unknown /proc/88/root (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 88 root txt unknown /proc/88/exe (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 88 root NOFD /proc/88/fd (opendir: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 93 root cwd unknown /proc/93/cwd (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 93 root rtd unknown /proc/93/root (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 93 root txt unknown /proc/93/exe (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 93 root NOFD /proc/93/fd (opendir: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_0 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-03-16T12:24:12.176 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_0 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_0 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_0 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 95 root cwd unknown /proc/95/cwd (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 95 root rtd unknown /proc/95/root (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 95 root txt unknown /proc/95/exe (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 95 root NOFD /proc/95/fd (opendir: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_1 96 root cwd unknown /proc/96/cwd (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_1 96 root rtd unknown /proc/96/root (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_1 96 root txt unknown /proc/96/exe (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_1 96 root NOFD /proc/96/fd (opendir: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 97 root cwd unknown /proc/97/cwd (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 97 root rtd unknown /proc/97/root (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 97 root txt unknown /proc/97/exe (readlink: Permission denied) 2024-03-16T12:24:12.177 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 97 root NOFD /proc/97/fd (opendir: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_2 98 root cwd unknown /proc/98/cwd (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_2 98 root rtd unknown /proc/98/root (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_2 98 root txt unknown /proc/98/exe (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_2 98 root NOFD /proc/98/fd (opendir: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 99 root cwd unknown /proc/99/cwd (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 99 root rtd unknown /proc/99/root (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 99 root txt unknown /proc/99/exe (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 99 root NOFD /proc/99/fd (opendir: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_3 100 root cwd unknown /proc/100/cwd (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_3 100 root rtd unknown /proc/100/root (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_3 100 root txt unknown /proc/100/exe (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_3 100 root NOFD /proc/100/fd (opendir: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 101 root cwd unknown /proc/101/cwd (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 101 root rtd unknown /proc/101/root (readlink: Permission denied) 2024-03-16T12:24:12.178 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 101 root txt unknown /proc/101/exe (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 101 root NOFD /proc/101/fd (opendir: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_4 105 root cwd unknown /proc/105/cwd (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_4 105 root rtd unknown /proc/105/root (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_4 105 root txt unknown /proc/105/exe (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_4 105 root NOFD /proc/105/fd (opendir: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 106 root cwd unknown /proc/106/cwd (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 106 root rtd unknown /proc/106/root (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 106 root txt unknown /proc/106/exe (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 106 root NOFD /proc/106/fd (opendir: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_5 107 root cwd unknown /proc/107/cwd (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_5 107 root rtd unknown /proc/107/root (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_5 107 root txt unknown /proc/107/exe (readlink: Permission denied) 2024-03-16T12:24:12.179 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_5 107 root NOFD /proc/107/fd (opendir: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_6 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_6 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_6 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_6 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_7 111 root cwd unknown /proc/111/cwd (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_7 111 root rtd unknown /proc/111/root (readlink: Permission denied) 2024-03-16T12:24:12.180 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_7 111 root txt unknown /proc/111/exe (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_7 111 root NOFD /proc/111/fd (opendir: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_8 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_8 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_8 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_8 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 114 root cwd unknown /proc/114/cwd (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 114 root rtd unknown /proc/114/root (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 114 root txt unknown /proc/114/exe (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 114 root NOFD /proc/114/fd (opendir: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_9 115 root cwd unknown /proc/115/cwd (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_9 115 root rtd unknown /proc/115/root (readlink: Permission denied) 2024-03-16T12:24:12.181 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_9 115 root txt unknown /proc/115/exe (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:scsi_eh_9 115 root NOFD /proc/115/fd (opendir: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 116 root cwd unknown /proc/116/cwd (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 116 root rtd unknown /proc/116/root (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 116 root txt unknown /proc/116/exe (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 116 root NOFD /proc/116/fd (opendir: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 117 root cwd unknown /proc/117/cwd (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 117 root rtd unknown /proc/117/root (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 117 root txt unknown /proc/117/exe (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 117 root NOFD /proc/117/fd (opendir: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 118 root cwd unknown /proc/118/cwd (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 118 root rtd unknown /proc/118/root (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 118 root txt unknown /proc/118/exe (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 118 root NOFD /proc/118/fd (opendir: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 119 root cwd unknown /proc/119/cwd (readlink: Permission denied) 2024-03-16T12:24:12.182 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 119 root rtd unknown /proc/119/root (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 119 root txt unknown /proc/119/exe (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 119 root NOFD /proc/119/fd (opendir: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 121 root cwd unknown /proc/121/cwd (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 121 root rtd unknown /proc/121/root (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 121 root txt unknown /proc/121/exe (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 121 root NOFD /proc/121/fd (opendir: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 130 root cwd unknown /proc/130/cwd (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 130 root rtd unknown /proc/130/root (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 130 root txt unknown /proc/130/exe (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 130 root NOFD /proc/130/fd (opendir: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 131 root cwd unknown /proc/131/cwd (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 131 root rtd unknown /proc/131/root (readlink: Permission denied) 2024-03-16T12:24:12.183 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 131 root txt unknown /proc/131/exe (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 131 root NOFD /proc/131/fd (opendir: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 132 root cwd unknown /proc/132/cwd (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 132 root rtd unknown /proc/132/root (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 132 root txt unknown /proc/132/exe (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 132 root NOFD /proc/132/fd (opendir: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 133 root cwd unknown /proc/133/cwd (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 133 root rtd unknown /proc/133/root (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 133 root txt unknown /proc/133/exe (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 133 root NOFD /proc/133/fd (opendir: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 134 root cwd unknown /proc/134/cwd (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 134 root rtd unknown /proc/134/root (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 134 root txt unknown /proc/134/exe (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 134 root NOFD /proc/134/fd (opendir: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 141 root cwd unknown /proc/141/cwd (readlink: Permission denied) 2024-03-16T12:24:12.184 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 141 root rtd unknown /proc/141/root (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 141 root txt unknown /proc/141/exe (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 141 root NOFD /proc/141/fd (opendir: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 211 root cwd unknown /proc/211/cwd (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 211 root rtd unknown /proc/211/root (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 211 root txt unknown /proc/211/exe (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 211 root NOFD /proc/211/fd (opendir: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 252 root cwd unknown /proc/252/cwd (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 252 root rtd unknown /proc/252/root (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 252 root txt unknown /proc/252/exe (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 252 root NOFD /proc/252/fd (opendir: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 257 root cwd unknown /proc/257/cwd (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 257 root rtd unknown /proc/257/root (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 257 root txt unknown /proc/257/exe (readlink: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 257 root NOFD /proc/257/fd (opendir: Permission denied) 2024-03-16T12:24:12.185 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 262 root cwd unknown /proc/262/cwd (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 262 root rtd unknown /proc/262/root (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 262 root txt unknown /proc/262/exe (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 262 root NOFD /proc/262/fd (opendir: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 267 root cwd unknown /proc/267/cwd (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 267 root rtd unknown /proc/267/root (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 267 root txt unknown /proc/267/exe (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 267 root NOFD /proc/267/fd (opendir: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 402 root cwd unknown /proc/402/cwd (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 402 root rtd unknown /proc/402/root (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 402 root txt unknown /proc/402/exe (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 402 root NOFD /proc/402/fd (opendir: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 406 root cwd unknown /proc/406/cwd (readlink: Permission denied) 2024-03-16T12:24:12.186 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 406 root rtd unknown /proc/406/root (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 406 root txt unknown /proc/406/exe (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 406 root NOFD /proc/406/fd (opendir: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 407 root cwd unknown /proc/407/cwd (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 407 root rtd unknown /proc/407/root (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 407 root txt unknown /proc/407/exe (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 407 root NOFD /proc/407/fd (opendir: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 432 root cwd unknown /proc/432/cwd (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 432 root rtd unknown /proc/432/root (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 432 root txt unknown /proc/432/exe (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 432 root NOFD /proc/432/fd (opendir: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 436 root cwd unknown /proc/436/cwd (readlink: Permission denied) 2024-03-16T12:24:12.187 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 436 root rtd unknown /proc/436/root (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 436 root txt unknown /proc/436/exe (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 436 root NOFD /proc/436/fd (opendir: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:jbd2/sda1 448 root cwd unknown /proc/448/cwd (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:jbd2/sda1 448 root rtd unknown /proc/448/root (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:jbd2/sda1 448 root txt unknown /proc/448/exe (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:jbd2/sda1 448 root NOFD /proc/448/fd (opendir: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 449 root cwd unknown /proc/449/cwd (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 449 root rtd unknown /proc/449/root (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 449 root txt unknown /proc/449/exe (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 449 root NOFD /proc/449/fd (opendir: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-j 525 root cwd unknown /proc/525/cwd (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-j 525 root rtd unknown /proc/525/root (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-j 525 root txt unknown /proc/525/exe (readlink: Permission denied) 2024-03-16T12:24:12.188 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-j 525 root NOFD /proc/525/fd (opendir: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:ext4lazyi 531 root cwd unknown /proc/531/cwd (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:ext4lazyi 531 root rtd unknown /proc/531/root (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:ext4lazyi 531 root txt unknown /proc/531/exe (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:ext4lazyi 531 root NOFD /proc/531/fd (opendir: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-u 544 root cwd unknown /proc/544/cwd (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-u 544 root rtd unknown /proc/544/root (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-u 544 root txt unknown /proc/544/exe (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-u 544 root NOFD /proc/544/fd (opendir: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 593 root cwd unknown /proc/593/cwd (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 593 root rtd unknown /proc/593/root (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 593 root txt unknown /proc/593/exe (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 593 root NOFD /proc/593/fd (opendir: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kipmi0 609 root cwd unknown /proc/609/cwd (readlink: Permission denied) 2024-03-16T12:24:12.189 INFO:tasks.workunit.client.0.smithi098.stdout:kipmi0 609 root rtd unknown /proc/609/root (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kipmi0 609 root txt unknown /proc/609/exe (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kipmi0 609 root NOFD /proc/609/fd (opendir: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 617 root cwd unknown /proc/617/cwd (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 617 root rtd unknown /proc/617/root (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 617 root txt unknown /proc/617/exe (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 617 root NOFD /proc/617/fd (opendir: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 618 root cwd unknown /proc/618/cwd (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 618 root rtd unknown /proc/618/root (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 618 root txt unknown /proc/618/exe (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 618 root NOFD /proc/618/fd (opendir: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:rpcbind 619 rpc cwd unknown /proc/619/cwd (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:rpcbind 619 rpc rtd unknown /proc/619/root (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:rpcbind 619 rpc txt unknown /proc/619/exe (readlink: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:rpcbind 619 rpc NOFD /proc/619/fd (opendir: Permission denied) 2024-03-16T12:24:12.190 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 root cwd unknown /proc/620/cwd (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 root rtd unknown /proc/620/root (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 root txt unknown /proc/620/exe (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 root NOFD /proc/620/fd (opendir: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 621 auditd root cwd unknown /proc/620/task/621/cwd (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 621 auditd root rtd unknown /proc/620/task/621/root (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 621 auditd root txt unknown /proc/620/task/621/exe (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 621 auditd root NOFD /proc/620/task/621/fd (opendir: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 624 auditd root cwd unknown /proc/620/task/624/cwd (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 624 auditd root rtd unknown /proc/620/task/624/root (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 624 auditd root txt unknown /proc/620/task/624/exe (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:auditd 620 624 auditd root NOFD /proc/620/task/624/fd (opendir: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:sedispatc 623 root cwd unknown /proc/623/cwd (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:sedispatc 623 root rtd unknown /proc/623/root (readlink: Permission denied) 2024-03-16T12:24:12.191 INFO:tasks.workunit.client.0.smithi098.stdout:sedispatc 623 root txt unknown /proc/623/exe (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:sedispatc 623 root NOFD /proc/623/fd (opendir: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 660 dbus cwd unknown /proc/660/cwd (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 660 dbus rtd unknown /proc/660/root (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 660 dbus txt unknown /proc/660/exe (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 660 dbus NOFD /proc/660/fd (opendir: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 661 dbus cwd unknown /proc/661/cwd (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 661 dbus rtd unknown /proc/661/root (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 661 dbus txt unknown /proc/661/exe (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:dbus-brok 661 dbus NOFD /proc/661/fd (opendir: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 root cwd unknown /proc/662/cwd (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 root rtd unknown /proc/662/root (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 root txt unknown /proc/662/exe (readlink: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 root NOFD /proc/662/fd (opendir: Permission denied) 2024-03-16T12:24:12.192 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 679 gmain root cwd unknown /proc/662/task/679/cwd (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 679 gmain root rtd unknown /proc/662/task/679/root (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 679 gmain root txt unknown /proc/662/task/679/exe (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 679 gmain root NOFD /proc/662/task/679/fd (opendir: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 680 gdbus root cwd unknown /proc/662/task/680/cwd (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 680 gdbus root rtd unknown /proc/662/task/680/root (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 680 gdbus root txt unknown /proc/662/task/680/exe (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:NetworkMa 662 680 gdbus root NOFD /proc/662/task/680/fd (opendir: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 root cwd unknown /proc/665/cwd (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 root rtd unknown /proc/665/root (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 root txt unknown /proc/665/exe (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 root NOFD /proc/665/fd (opendir: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 669 gmain root cwd unknown /proc/665/task/669/cwd (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 669 gmain root rtd unknown /proc/665/task/669/root (readlink: Permission denied) 2024-03-16T12:24:12.193 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 669 gmain root txt unknown /proc/665/task/669/exe (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:irqbalanc 665 669 gmain root NOFD /proc/665/task/669/fd (opendir: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:lsmd 666 libstoragemgmt cwd unknown /proc/666/cwd (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:lsmd 666 libstoragemgmt rtd unknown /proc/666/root (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:lsmd 666 libstoragemgmt txt unknown /proc/666/exe (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:lsmd 666 libstoragemgmt NOFD /proc/666/fd (opendir: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 polkitd cwd unknown /proc/667/cwd (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 polkitd rtd unknown /proc/667/root (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 polkitd txt unknown /proc/667/exe (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 polkitd NOFD /proc/667/fd (opendir: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 681 gmain polkitd cwd unknown /proc/667/task/681/cwd (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 681 gmain polkitd rtd unknown /proc/667/task/681/root (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 681 gmain polkitd txt unknown /proc/667/task/681/exe (readlink: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 681 gmain polkitd NOFD /proc/667/task/681/fd (opendir: Permission denied) 2024-03-16T12:24:12.194 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 683 gdbus polkitd cwd unknown /proc/667/task/683/cwd (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 683 gdbus polkitd rtd unknown /proc/667/task/683/root (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 683 gdbus polkitd txt unknown /proc/667/task/683/exe (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 683 gdbus polkitd NOFD /proc/667/task/683/fd (opendir: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 684 JS\x20Hel polkitd cwd unknown /proc/667/task/684/cwd (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 684 JS\x20Hel polkitd rtd unknown /proc/667/task/684/root (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 684 JS\x20Hel polkitd txt unknown /proc/667/task/684/exe (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 684 JS\x20Hel polkitd NOFD /proc/667/task/684/fd (opendir: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 685 JS\x20Hel polkitd cwd unknown /proc/667/task/685/cwd (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 685 JS\x20Hel polkitd rtd unknown /proc/667/task/685/root (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 685 JS\x20Hel polkitd txt unknown /proc/667/task/685/exe (readlink: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 685 JS\x20Hel polkitd NOFD /proc/667/task/685/fd (opendir: Permission denied) 2024-03-16T12:24:12.195 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 686 JS\x20Hel polkitd cwd unknown /proc/667/task/686/cwd (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 686 JS\x20Hel polkitd rtd unknown /proc/667/task/686/root (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 686 JS\x20Hel polkitd txt unknown /proc/667/task/686/exe (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 686 JS\x20Hel polkitd NOFD /proc/667/task/686/fd (opendir: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 687 JS\x20Hel polkitd cwd unknown /proc/667/task/687/cwd (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 687 JS\x20Hel polkitd rtd unknown /proc/667/task/687/root (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 687 JS\x20Hel polkitd txt unknown /proc/667/task/687/exe (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 687 JS\x20Hel polkitd NOFD /proc/667/task/687/fd (opendir: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 688 JS\x20Hel polkitd cwd unknown /proc/667/task/688/cwd (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 688 JS\x20Hel polkitd rtd unknown /proc/667/task/688/root (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 688 JS\x20Hel polkitd txt unknown /proc/667/task/688/exe (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 688 JS\x20Hel polkitd NOFD /proc/667/task/688/fd (opendir: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 689 JS\x20Hel polkitd cwd unknown /proc/667/task/689/cwd (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 689 JS\x20Hel polkitd rtd unknown /proc/667/task/689/root (readlink: Permission denied) 2024-03-16T12:24:12.196 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 689 JS\x20Hel polkitd txt unknown /proc/667/task/689/exe (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 689 JS\x20Hel polkitd NOFD /proc/667/task/689/fd (opendir: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 690 JS\x20Hel polkitd cwd unknown /proc/667/task/690/cwd (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 690 JS\x20Hel polkitd rtd unknown /proc/667/task/690/root (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 690 JS\x20Hel polkitd txt unknown /proc/667/task/690/exe (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 690 JS\x20Hel polkitd NOFD /proc/667/task/690/fd (opendir: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 691 JS\x20Hel polkitd cwd unknown /proc/667/task/691/cwd (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 691 JS\x20Hel polkitd rtd unknown /proc/667/task/691/root (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 691 JS\x20Hel polkitd txt unknown /proc/667/task/691/exe (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 691 JS\x20Hel polkitd NOFD /proc/667/task/691/fd (opendir: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 693 polkitd polkitd cwd unknown /proc/667/task/693/cwd (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 693 polkitd polkitd rtd unknown /proc/667/task/693/root (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 693 polkitd polkitd txt unknown /proc/667/task/693/exe (readlink: Permission denied) 2024-03-16T12:24:12.197 INFO:tasks.workunit.client.0.smithi098.stdout:polkitd 667 693 polkitd polkitd NOFD /proc/667/task/693/fd (opendir: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-l 671 root cwd unknown /proc/671/cwd (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-l 671 root rtd unknown /proc/671/root (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-l 671 root txt unknown /proc/671/exe (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:systemd-l 671 root NOFD /proc/671/fd (opendir: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 726 root cwd unknown /proc/726/cwd (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 726 root rtd unknown /proc/726/root (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 726 root txt unknown /proc/726/exe (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 726 root NOFD /proc/726/fd (opendir: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 root cwd unknown /proc/727/cwd (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 root rtd unknown /proc/727/root (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 root txt unknown /proc/727/exe (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 root NOFD /proc/727/fd (opendir: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 812 tuned root cwd unknown /proc/727/task/812/cwd (readlink: Permission denied) 2024-03-16T12:24:12.198 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 812 tuned root rtd unknown /proc/727/task/812/root (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 812 tuned root txt unknown /proc/727/task/812/exe (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:tuned 727 812 tuned root NOFD /proc/727/task/812/fd (opendir: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 root cwd unknown /proc/731/cwd (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 root rtd unknown /proc/731/root (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 root txt unknown /proc/731/exe (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 root NOFD /proc/731/fd (opendir: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 732 gssproxy root cwd unknown /proc/731/task/732/cwd (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 732 gssproxy root rtd unknown /proc/731/task/732/root (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 732 gssproxy root txt unknown /proc/731/task/732/exe (readlink: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 732 gssproxy root NOFD /proc/731/task/732/fd (opendir: Permission denied) 2024-03-16T12:24:12.199 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 733 gssproxy root cwd unknown /proc/731/task/733/cwd (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 733 gssproxy root rtd unknown /proc/731/task/733/root (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 733 gssproxy root txt unknown /proc/731/task/733/exe (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 733 gssproxy root NOFD /proc/731/task/733/fd (opendir: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 734 gssproxy root cwd unknown /proc/731/task/734/cwd (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 734 gssproxy root rtd unknown /proc/731/task/734/root (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 734 gssproxy root txt unknown /proc/731/task/734/exe (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 734 gssproxy root NOFD /proc/731/task/734/fd (opendir: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 735 gssproxy root cwd unknown /proc/731/task/735/cwd (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 735 gssproxy root rtd unknown /proc/731/task/735/root (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 735 gssproxy root txt unknown /proc/731/task/735/exe (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 735 gssproxy root NOFD /proc/731/task/735/fd (opendir: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 736 gssproxy root cwd unknown /proc/731/task/736/cwd (readlink: Permission denied) 2024-03-16T12:24:12.200 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 736 gssproxy root rtd unknown /proc/731/task/736/root (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 736 gssproxy root txt unknown /proc/731/task/736/exe (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:gssproxy 731 736 gssproxy root NOFD /proc/731/task/736/fd (opendir: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:atd 738 root cwd unknown /proc/738/cwd (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:atd 738 root rtd unknown /proc/738/root (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:atd 738 root txt unknown /proc/738/exe (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:atd 738 root NOFD /proc/738/fd (opendir: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:crond 739 root cwd unknown /proc/739/cwd (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:crond 739 root rtd unknown /proc/739/root (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:crond 739 root txt unknown /proc/739/exe (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:crond 739 root NOFD /proc/739/fd (opendir: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:nrpe 834 nrpe cwd unknown /proc/834/cwd (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:nrpe 834 nrpe rtd unknown /proc/834/root (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:nrpe 834 nrpe txt unknown /proc/834/exe (readlink: Permission denied) 2024-03-16T12:24:12.201 INFO:tasks.workunit.client.0.smithi098.stdout:nrpe 834 nrpe NOFD /proc/834/fd (opendir: Permission denied) 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu cwd DIR 8,1 4096 2 / 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 1167 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 10007 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 1705 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-03-16T12:24:12.202 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 10126 /usr/lib64/libblkid.so.1.1.0 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 10156 /usr/lib64/libmount.so.1.1.0 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-03-16T12:24:12.203 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu DEL REG 8,1 9806 /usr/lib64/libacl.so.1.1.2301 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 1u unix 0x0000000000000000 0t0 2405 type=STREAM (CONNECTED) 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 2u unix 0x0000000000000000 0t0 2405 type=STREAM (CONNECTED) 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 3u unix 0x0000000000000000 0t0 2437 type=DGRAM (CONNECTED) 2024-03-16T12:24:12.204 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 4u a_inode 0,14 0 50 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 5u a_inode 0,14 0 50 [signalfd] 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 6r a_inode 0,14 0 50 inotify 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 7r DIR 0,25 0 3353 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 8u a_inode 0,14 0 50 [timerfd] 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 9u a_inode 0,14 0 50 [eventpoll:10,13] 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 10r REG 0,20 0 2424 /proc/1030/mountinfo 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 11r a_inode 0,14 0 50 inotify 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 12u unix 0x0000000000000000 0t0 7846 /run/user/1000/bus type=STREAM (LISTEN) 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 13r a_inode 0,14 0 50 inotify 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 14r REG 0,20 0 4026532078 /proc/swaps 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 15u netlink 0t0 2442 KOBJECT_UEVENT 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 16u unix 0x0000000000000000 0t0 2447 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 17u unix 0x0000000000000000 0t0 2448 type=DGRAM (CONNECTED) 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 18u unix 0x0000000000000000 0t0 2449 type=DGRAM (CONNECTED) 2024-03-16T12:24:12.205 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 19u unix 0x0000000000000000 0t0 2450 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 20u unix 0x0000000000000000 0t0 2452 type=STREAM (CONNECTED) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 1030 ubuntu 21u a_inode 0,14 0 50 [timerfd] 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 1057 ubuntu cwd unknown /proc/1057/cwd (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 1057 ubuntu rtd unknown /proc/1057/root (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 1057 ubuntu txt unknown /proc/1057/exe (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 1057 ubuntu NOFD /proc/1057/fd (opendir: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1206 root cwd unknown /proc/1206/cwd (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1206 root rtd unknown /proc/1206/root (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1206 root txt unknown /proc/1206/exe (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1206 root NOFD /proc/1206/fd (opendir: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1207 root cwd unknown /proc/1207/cwd (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1207 root rtd unknown /proc/1207/root (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1207 root txt unknown /proc/1207/exe (readlink: Permission denied) 2024-03-16T12:24:12.206 INFO:tasks.workunit.client.0.smithi098.stdout:agetty 1207 root NOFD /proc/1207/fd (opendir: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 root cwd unknown /proc/2034/cwd (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 root rtd unknown /proc/2034/root (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 root txt unknown /proc/2034/exe (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 root NOFD /proc/2034/fd (opendir: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2048 in:imjour root cwd unknown /proc/2034/task/2048/cwd (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2048 in:imjour root rtd unknown /proc/2034/task/2048/root (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2048 in:imjour root txt unknown /proc/2034/task/2048/exe (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2048 in:imjour root NOFD /proc/2034/task/2048/fd (opendir: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2049 rs:main root cwd unknown /proc/2034/task/2049/cwd (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2049 rs:main root rtd unknown /proc/2034/task/2049/root (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2049 rs:main root txt unknown /proc/2034/task/2049/exe (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:rsyslogd 2034 2049 rs:main root NOFD /proc/2034/task/2049/fd (opendir: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:anacron 5970 root cwd unknown /proc/5970/cwd (readlink: Permission denied) 2024-03-16T12:24:12.207 INFO:tasks.workunit.client.0.smithi098.stdout:anacron 5970 root rtd unknown /proc/5970/root (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:anacron 5970 root txt unknown /proc/5970/exe (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:anacron 5970 root NOFD /proc/5970/fd (opendir: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:smartd 15287 root cwd unknown /proc/15287/cwd (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:smartd 15287 root rtd unknown /proc/15287/root (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:smartd 15287 root txt unknown /proc/15287/exe (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:smartd 15287 root NOFD /proc/15287/fd (opendir: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20651 root cwd unknown /proc/20651/cwd (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20651 root rtd unknown /proc/20651/root (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20651 root txt unknown /proc/20651/exe (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20651 root NOFD /proc/20651/fd (opendir: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20681 root cwd unknown /proc/20681/cwd (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20681 root rtd unknown /proc/20681/root (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20681 root txt unknown /proc/20681/exe (readlink: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20681 root NOFD /proc/20681/fd (opendir: Permission denied) 2024-03-16T12:24:12.208 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20710 root cwd unknown /proc/20710/cwd (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20710 root rtd unknown /proc/20710/root (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20710 root txt unknown /proc/20710/exe (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20710 root NOFD /proc/20710/fd (opendir: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20739 root cwd unknown /proc/20739/cwd (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20739 root rtd unknown /proc/20739/root (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20739 root txt unknown /proc/20739/exe (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20739 root NOFD /proc/20739/fd (opendir: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20768 root cwd unknown /proc/20768/cwd (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20768 root rtd unknown /proc/20768/root (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20768 root txt unknown /proc/20768/exe (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20768 root NOFD /proc/20768/fd (opendir: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20932 root cwd unknown /proc/20932/cwd (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20932 root rtd unknown /proc/20932/root (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20932 root txt unknown /proc/20932/exe (readlink: Permission denied) 2024-03-16T12:24:12.209 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20932 root NOFD /proc/20932/fd (opendir: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20933 root cwd unknown /proc/20933/cwd (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20933 root rtd unknown /proc/20933/root (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20933 root txt unknown /proc/20933/exe (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20933 root NOFD /proc/20933/fd (opendir: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20934 root cwd unknown /proc/20934/cwd (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20934 root rtd unknown /proc/20934/root (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20934 root txt unknown /proc/20934/exe (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20934 root NOFD /proc/20934/fd (opendir: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20935 root cwd unknown /proc/20935/cwd (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20935 root rtd unknown /proc/20935/root (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20935 root txt unknown /proc/20935/exe (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20935 root NOFD /proc/20935/fd (opendir: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20936 root cwd unknown /proc/20936/cwd (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20936 root rtd unknown /proc/20936/root (readlink: Permission denied) 2024-03-16T12:24:12.210 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20936 root txt unknown /proc/20936/exe (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20936 root NOFD /proc/20936/fd (opendir: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20937 root cwd unknown /proc/20937/cwd (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20937 root rtd unknown /proc/20937/root (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20937 root txt unknown /proc/20937/exe (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20937 root NOFD /proc/20937/fd (opendir: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20938 root cwd unknown /proc/20938/cwd (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20938 root rtd unknown /proc/20938/root (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20938 root txt unknown /proc/20938/exe (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20938 root NOFD /proc/20938/fd (opendir: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20939 root cwd unknown /proc/20939/cwd (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20939 root rtd unknown /proc/20939/root (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20939 root txt unknown /proc/20939/exe (readlink: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20939 root NOFD /proc/20939/fd (opendir: Permission denied) 2024-03-16T12:24:12.211 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20940 root cwd unknown /proc/20940/cwd (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20940 root rtd unknown /proc/20940/root (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20940 root txt unknown /proc/20940/exe (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 20940 root NOFD /proc/20940/fd (opendir: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:xfsaild/d 20941 root cwd unknown /proc/20941/cwd (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:xfsaild/d 20941 root rtd unknown /proc/20941/root (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:xfsaild/d 20941 root txt unknown /proc/20941/exe (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:xfsaild/d 20941 root NOFD /proc/20941/fd (opendir: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22876 root cwd unknown /proc/22876/cwd (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22876 root rtd unknown /proc/22876/root (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22876 root txt unknown /proc/22876/exe (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22876 root NOFD /proc/22876/fd (opendir: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22879 ubuntu cwd unknown /proc/22879/cwd (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22879 ubuntu rtd unknown /proc/22879/root (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22879 ubuntu txt unknown /proc/22879/exe (readlink: Permission denied) 2024-03-16T12:24:12.212 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 22879 ubuntu NOFD /proc/22879/fd (opendir: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:chronyd 22939 chrony cwd unknown /proc/22939/cwd (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:chronyd 22939 chrony rtd unknown /proc/22939/root (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:chronyd 22939 chrony txt unknown /proc/22939/exe (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:chronyd 22939 chrony NOFD /proc/22939/fd (opendir: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-cras 23299 ceph cwd unknown /proc/23299/cwd (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-cras 23299 ceph rtd unknown /proc/23299/root (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-cras 23299 ceph txt unknown /proc/23299/exe (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-cras 23299 ceph NOFD /proc/23299/fd (opendir: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24129 root cwd unknown /proc/24129/cwd (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24129 root rtd unknown /proc/24129/root (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24129 root txt unknown /proc/24129/exe (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24129 root NOFD /proc/24129/fd (opendir: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24197 root cwd unknown /proc/24197/cwd (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24197 root rtd unknown /proc/24197/root (readlink: Permission denied) 2024-03-16T12:24:12.213 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24197 root txt unknown /proc/24197/exe (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/4 24197 root NOFD /proc/24197/fd (opendir: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24418 root cwd unknown /proc/24418/cwd (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24418 root rtd unknown /proc/24418/root (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24418 root txt unknown /proc/24418/exe (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24418 root NOFD /proc/24418/fd (opendir: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24424 root cwd unknown /proc/24424/cwd (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24424 root rtd unknown /proc/24424/root (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24424 root txt unknown /proc/24424/exe (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 24424 root NOFD /proc/24424/fd (opendir: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 25114 root cwd unknown /proc/25114/cwd (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 25114 root rtd unknown /proc/25114/root (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 25114 root txt unknown /proc/25114/exe (readlink: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 25114 root NOFD /proc/25114/fd (opendir: Permission denied) 2024-03-16T12:24:12.214 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 25178 root cwd unknown /proc/25178/cwd (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 25178 root rtd unknown /proc/25178/root (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 25178 root txt unknown /proc/25178/exe (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 25178 root NOFD /proc/25178/fd (opendir: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30630 root cwd unknown /proc/30630/cwd (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30630 root rtd unknown /proc/30630/root (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30630 root txt unknown /proc/30630/exe (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30630 root NOFD /proc/30630/fd (opendir: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30646 root cwd unknown /proc/30646/cwd (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30646 root rtd unknown /proc/30646/root (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30646 root txt unknown /proc/30646/exe (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 30646 root NOFD /proc/30646/fd (opendir: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30663 root cwd unknown /proc/30663/cwd (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30663 root rtd unknown /proc/30663/root (readlink: Permission denied) 2024-03-16T12:24:12.215 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30663 root txt unknown /proc/30663/exe (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30663 root NOFD /proc/30663/fd (opendir: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30679 root cwd unknown /proc/30679/cwd (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30679 root rtd unknown /proc/30679/root (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30679 root txt unknown /proc/30679/exe (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 30679 root NOFD /proc/30679/fd (opendir: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31628 root cwd unknown /proc/31628/cwd (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31628 root rtd unknown /proc/31628/root (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31628 root txt unknown /proc/31628/exe (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31628 root NOFD /proc/31628/fd (opendir: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31630 root cwd unknown /proc/31630/cwd (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31630 root rtd unknown /proc/31630/root (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31630 root txt unknown /proc/31630/exe (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31630 root NOFD /proc/31630/fd (opendir: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 ceph cwd unknown /proc/31633/cwd (readlink: Permission denied) 2024-03-16T12:24:12.216 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 ceph rtd unknown /proc/31633/root (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 ceph txt unknown /proc/31633/exe (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 ceph NOFD /proc/31633/fd (opendir: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31634 log ceph cwd unknown /proc/31633/task/31634/cwd (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31634 log ceph rtd unknown /proc/31633/task/31634/root (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31634 log ceph txt unknown /proc/31633/task/31634/exe (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31634 log ceph NOFD /proc/31633/task/31634/fd (opendir: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31635 service ceph cwd unknown /proc/31633/task/31635/cwd (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31635 service ceph rtd unknown /proc/31633/task/31635/root (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31635 service ceph txt unknown /proc/31633/task/31635/exe (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31635 service ceph NOFD /proc/31633/task/31635/fd (opendir: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31636 admin_soc ceph cwd unknown /proc/31633/task/31636/cwd (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31636 admin_soc ceph rtd unknown /proc/31633/task/31636/root (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31636 admin_soc ceph txt unknown /proc/31633/task/31636/exe (readlink: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31636 admin_soc ceph NOFD /proc/31633/task/31636/fd (opendir: Permission denied) 2024-03-16T12:24:12.217 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31637 signal_ha ceph cwd unknown /proc/31633/task/31637/cwd (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31637 signal_ha ceph rtd unknown /proc/31633/task/31637/root (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31637 signal_ha ceph txt unknown /proc/31633/task/31637/exe (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31637 signal_ha ceph NOFD /proc/31633/task/31637/fd (opendir: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31638 rocksdb:l ceph cwd unknown /proc/31633/task/31638/cwd (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31638 rocksdb:l ceph rtd unknown /proc/31633/task/31638/root (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31638 rocksdb:l ceph txt unknown /proc/31633/task/31638/exe (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31638 rocksdb:l ceph NOFD /proc/31633/task/31638/fd (opendir: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31639 rocksdb:h ceph cwd unknown /proc/31633/task/31639/cwd (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31639 rocksdb:h ceph rtd unknown /proc/31633/task/31639/root (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31639 rocksdb:h ceph txt unknown /proc/31633/task/31639/exe (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31639 rocksdb:h ceph NOFD /proc/31633/task/31639/fd (opendir: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31655 ceph-mon ceph cwd unknown /proc/31633/task/31655/cwd (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31655 ceph-mon ceph rtd unknown /proc/31633/task/31655/root (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31655 ceph-mon ceph txt unknown /proc/31633/task/31655/exe (readlink: Permission denied) 2024-03-16T12:24:12.218 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31655 ceph-mon ceph NOFD /proc/31633/task/31655/fd (opendir: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31656 fn_monsto ceph cwd unknown /proc/31633/task/31656/cwd (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31656 fn_monsto ceph rtd unknown /proc/31633/task/31656/root (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31656 fn_monsto ceph txt unknown /proc/31633/task/31656/exe (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31656 fn_monsto ceph NOFD /proc/31633/task/31656/fd (opendir: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31657 msgr-work ceph cwd unknown /proc/31633/task/31657/cwd (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31657 msgr-work ceph rtd unknown /proc/31633/task/31657/root (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31657 msgr-work ceph txt unknown /proc/31633/task/31657/exe (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31657 msgr-work ceph NOFD /proc/31633/task/31657/fd (opendir: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31658 msgr-work ceph cwd unknown /proc/31633/task/31658/cwd (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31658 msgr-work ceph rtd unknown /proc/31633/task/31658/root (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31658 msgr-work ceph txt unknown /proc/31633/task/31658/exe (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31658 msgr-work ceph NOFD /proc/31633/task/31658/fd (opendir: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31659 msgr-work ceph cwd unknown /proc/31633/task/31659/cwd (readlink: Permission denied) 2024-03-16T12:24:12.219 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31659 msgr-work ceph rtd unknown /proc/31633/task/31659/root (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31659 msgr-work ceph txt unknown /proc/31633/task/31659/exe (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31659 msgr-work ceph NOFD /proc/31633/task/31659/fd (opendir: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31660 OpHistory ceph cwd unknown /proc/31633/task/31660/cwd (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31660 OpHistory ceph rtd unknown /proc/31633/task/31660/root (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31660 OpHistory ceph txt unknown /proc/31633/task/31660/exe (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31660 OpHistory ceph NOFD /proc/31633/task/31660/fd (opendir: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31661 fin ceph cwd unknown /proc/31633/task/31661/cwd (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31661 fin ceph rtd unknown /proc/31633/task/31661/root (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31661 fin ceph txt unknown /proc/31633/task/31661/exe (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31661 fin ceph NOFD /proc/31633/task/31661/fd (opendir: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31662 safe_time ceph cwd unknown /proc/31633/task/31662/cwd (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31662 safe_time ceph rtd unknown /proc/31633/task/31662/root (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31662 safe_time ceph txt unknown /proc/31633/task/31662/exe (readlink: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31662 safe_time ceph NOFD /proc/31633/task/31662/fd (opendir: Permission denied) 2024-03-16T12:24:12.220 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31663 cpu_tp ceph cwd unknown /proc/31633/task/31663/cwd (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31663 cpu_tp ceph rtd unknown /proc/31633/task/31663/root (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31663 cpu_tp ceph txt unknown /proc/31633/task/31663/exe (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31663 cpu_tp ceph NOFD /proc/31633/task/31663/fd (opendir: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31664 cpu_tp ceph cwd unknown /proc/31633/task/31664/cwd (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31664 cpu_tp ceph rtd unknown /proc/31633/task/31664/root (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31664 cpu_tp ceph txt unknown /proc/31633/task/31664/exe (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31664 cpu_tp ceph NOFD /proc/31633/task/31664/fd (opendir: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31665 cpu_tp ceph cwd unknown /proc/31633/task/31665/cwd (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31665 cpu_tp ceph rtd unknown /proc/31633/task/31665/root (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31665 cpu_tp ceph txt unknown /proc/31633/task/31665/exe (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31665 cpu_tp ceph NOFD /proc/31633/task/31665/fd (opendir: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31666 cpu_tp ceph cwd unknown /proc/31633/task/31666/cwd (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31666 cpu_tp ceph rtd unknown /proc/31633/task/31666/root (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31666 cpu_tp ceph txt unknown /proc/31633/task/31666/exe (readlink: Permission denied) 2024-03-16T12:24:12.221 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31666 cpu_tp ceph NOFD /proc/31633/task/31666/fd (opendir: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31667 ms_dispat ceph cwd unknown /proc/31633/task/31667/cwd (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31667 ms_dispat ceph rtd unknown /proc/31633/task/31667/root (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31667 ms_dispat ceph txt unknown /proc/31633/task/31667/exe (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31667 ms_dispat ceph NOFD /proc/31633/task/31667/fd (opendir: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31668 ms_local ceph cwd unknown /proc/31633/task/31668/cwd (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31668 ms_local ceph rtd unknown /proc/31633/task/31668/root (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31668 ms_local ceph txt unknown /proc/31633/task/31668/exe (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31668 ms_local ceph NOFD /proc/31633/task/31668/fd (opendir: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31669 safe_time ceph cwd unknown /proc/31633/task/31669/cwd (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31669 safe_time ceph rtd unknown /proc/31633/task/31669/root (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31669 safe_time ceph txt unknown /proc/31633/task/31669/exe (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31669 safe_time ceph NOFD /proc/31633/task/31669/fd (opendir: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31670 ms_dispat ceph cwd unknown /proc/31633/task/31670/cwd (readlink: Permission denied) 2024-03-16T12:24:12.222 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31670 ms_dispat ceph rtd unknown /proc/31633/task/31670/root (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31670 ms_dispat ceph txt unknown /proc/31633/task/31670/exe (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31670 ms_dispat ceph NOFD /proc/31633/task/31670/fd (opendir: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31671 ms_local ceph cwd unknown /proc/31633/task/31671/cwd (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31671 ms_local ceph rtd unknown /proc/31633/task/31671/root (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31671 ms_local ceph txt unknown /proc/31633/task/31671/exe (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 31671 ms_local ceph NOFD /proc/31633/task/31671/fd (opendir: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 68370 rstore_co ceph cwd unknown /proc/31633/task/68370/cwd (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 68370 rstore_co ceph rtd unknown /proc/31633/task/68370/root (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 68370 rstore_co ceph txt unknown /proc/31633/task/68370/exe (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mon 31633 68370 rstore_co ceph NOFD /proc/31633/task/68370/fd (opendir: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31872 root cwd unknown /proc/31872/cwd (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31872 root rtd unknown /proc/31872/root (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31872 root txt unknown /proc/31872/exe (readlink: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 31872 root NOFD /proc/31872/fd (opendir: Permission denied) 2024-03-16T12:24:12.223 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31874 root cwd unknown /proc/31874/cwd (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31874 root rtd unknown /proc/31874/root (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31874 root txt unknown /proc/31874/exe (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 31874 root NOFD /proc/31874/fd (opendir: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 ceph cwd unknown /proc/31876/cwd (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 ceph rtd unknown /proc/31876/root (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 ceph txt unknown /proc/31876/exe (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 ceph NOFD /proc/31876/fd (opendir: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34380 log ceph cwd unknown /proc/31876/task/34380/cwd (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34380 log ceph rtd unknown /proc/31876/task/34380/root (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34380 log ceph txt unknown /proc/31876/task/34380/exe (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34380 log ceph NOFD /proc/31876/task/34380/fd (opendir: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34382 msgr-work ceph cwd unknown /proc/31876/task/34382/cwd (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34382 msgr-work ceph rtd unknown /proc/31876/task/34382/root (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34382 msgr-work ceph txt unknown /proc/31876/task/34382/exe (readlink: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34382 msgr-work ceph NOFD /proc/31876/task/34382/fd (opendir: Permission denied) 2024-03-16T12:24:12.224 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34383 msgr-work ceph cwd unknown /proc/31876/task/34383/cwd (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34383 msgr-work ceph rtd unknown /proc/31876/task/34383/root (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34383 msgr-work ceph txt unknown /proc/31876/task/34383/exe (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34383 msgr-work ceph NOFD /proc/31876/task/34383/fd (opendir: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34384 msgr-work ceph cwd unknown /proc/31876/task/34384/cwd (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34384 msgr-work ceph rtd unknown /proc/31876/task/34384/root (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34384 msgr-work ceph txt unknown /proc/31876/task/34384/exe (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34384 msgr-work ceph NOFD /proc/31876/task/34384/fd (opendir: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34388 service ceph cwd unknown /proc/31876/task/34388/cwd (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34388 service ceph rtd unknown /proc/31876/task/34388/root (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34388 service ceph txt unknown /proc/31876/task/34388/exe (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34388 service ceph NOFD /proc/31876/task/34388/fd (opendir: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34389 admin_soc ceph cwd unknown /proc/31876/task/34389/cwd (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34389 admin_soc ceph rtd unknown /proc/31876/task/34389/root (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34389 admin_soc ceph txt unknown /proc/31876/task/34389/exe (readlink: Permission denied) 2024-03-16T12:24:12.225 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34389 admin_soc ceph NOFD /proc/31876/task/34389/fd (opendir: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34390 ceph_time ceph cwd unknown /proc/31876/task/34390/cwd (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34390 ceph_time ceph rtd unknown /proc/31876/task/34390/root (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34390 ceph_time ceph txt unknown /proc/31876/task/34390/exe (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34390 ceph_time ceph NOFD /proc/31876/task/34390/fd (opendir: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34391 fn_anonym ceph cwd unknown /proc/31876/task/34391/cwd (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34391 fn_anonym ceph rtd unknown /proc/31876/task/34391/root (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34391 fn_anonym ceph txt unknown /proc/31876/task/34391/exe (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34391 fn_anonym ceph NOFD /proc/31876/task/34391/fd (opendir: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34392 signal_ha ceph cwd unknown /proc/31876/task/34392/cwd (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34392 signal_ha ceph rtd unknown /proc/31876/task/34392/root (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34392 signal_ha ceph txt unknown /proc/31876/task/34392/exe (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34392 signal_ha ceph NOFD /proc/31876/task/34392/fd (opendir: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34393 mgrsb-fin ceph cwd unknown /proc/31876/task/34393/cwd (readlink: Permission denied) 2024-03-16T12:24:12.226 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34393 mgrsb-fin ceph rtd unknown /proc/31876/task/34393/root (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34393 mgrsb-fin ceph txt unknown /proc/31876/task/34393/exe (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34393 mgrsb-fin ceph NOFD /proc/31876/task/34393/fd (opendir: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34394 ms_dispat ceph cwd unknown /proc/31876/task/34394/cwd (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34394 ms_dispat ceph rtd unknown /proc/31876/task/34394/root (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34394 ms_dispat ceph txt unknown /proc/31876/task/34394/exe (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34394 ms_dispat ceph NOFD /proc/31876/task/34394/fd (opendir: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34395 ms_local ceph cwd unknown /proc/31876/task/34395/cwd (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34395 ms_local ceph rtd unknown /proc/31876/task/34395/root (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34395 ms_local ceph txt unknown /proc/31876/task/34395/exe (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34395 ms_local ceph NOFD /proc/31876/task/34395/fd (opendir: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34396 io_contex ceph cwd unknown /proc/31876/task/34396/cwd (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34396 io_contex ceph rtd unknown /proc/31876/task/34396/root (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34396 io_contex ceph txt unknown /proc/31876/task/34396/exe (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34396 io_contex ceph NOFD /proc/31876/task/34396/fd (opendir: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34397 io_contex ceph cwd unknown /proc/31876/task/34397/cwd (readlink: Permission denied) 2024-03-16T12:24:12.227 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34397 io_contex ceph rtd unknown /proc/31876/task/34397/root (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34397 io_contex ceph txt unknown /proc/31876/task/34397/exe (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34397 io_contex ceph NOFD /proc/31876/task/34397/fd (opendir: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34398 safe_time ceph cwd unknown /proc/31876/task/34398/cwd (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34398 safe_time ceph rtd unknown /proc/31876/task/34398/root (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34398 safe_time ceph txt unknown /proc/31876/task/34398/exe (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34398 safe_time ceph NOFD /proc/31876/task/34398/fd (opendir: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34399 safe_time ceph cwd unknown /proc/31876/task/34399/cwd (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34399 safe_time ceph rtd unknown /proc/31876/task/34399/root (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34399 safe_time ceph txt unknown /proc/31876/task/34399/exe (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34399 safe_time ceph NOFD /proc/31876/task/34399/fd (opendir: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34400 safe_time ceph cwd unknown /proc/31876/task/34400/cwd (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34400 safe_time ceph rtd unknown /proc/31876/task/34400/root (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34400 safe_time ceph txt unknown /proc/31876/task/34400/exe (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34400 safe_time ceph NOFD /proc/31876/task/34400/fd (opendir: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34401 fn_anonym ceph cwd unknown /proc/31876/task/34401/cwd (readlink: Permission denied) 2024-03-16T12:24:12.228 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34401 fn_anonym ceph rtd unknown /proc/31876/task/34401/root (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34401 fn_anonym ceph txt unknown /proc/31876/task/34401/exe (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34401 fn_anonym ceph NOFD /proc/31876/task/34401/fd (opendir: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34402 flusher ceph cwd unknown /proc/31876/task/34402/cwd (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34402 flusher ceph rtd unknown /proc/31876/task/34402/root (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34402 flusher ceph txt unknown /proc/31876/task/34402/exe (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34402 flusher ceph NOFD /proc/31876/task/34402/fd (opendir: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34403 safe_time ceph cwd unknown /proc/31876/task/34403/cwd (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34403 safe_time ceph rtd unknown /proc/31876/task/34403/root (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34403 safe_time ceph txt unknown /proc/31876/task/34403/exe (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34403 safe_time ceph NOFD /proc/31876/task/34403/fd (opendir: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34594 mgr-fin ceph cwd unknown /proc/31876/task/34594/cwd (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34594 mgr-fin ceph rtd unknown /proc/31876/task/34594/root (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34594 mgr-fin ceph txt unknown /proc/31876/task/34594/exe (readlink: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34594 mgr-fin ceph NOFD /proc/31876/task/34594/fd (opendir: Permission denied) 2024-03-16T12:24:12.229 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34595 ms_dispat ceph cwd unknown /proc/31876/task/34595/cwd (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34595 ms_dispat ceph rtd unknown /proc/31876/task/34595/root (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34595 ms_dispat ceph txt unknown /proc/31876/task/34595/exe (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34595 ms_dispat ceph NOFD /proc/31876/task/34595/fd (opendir: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34596 ms_local ceph cwd unknown /proc/31876/task/34596/cwd (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34596 ms_local ceph rtd unknown /proc/31876/task/34596/root (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34596 ms_local ceph txt unknown /proc/31876/task/34596/exe (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34596 ms_local ceph NOFD /proc/31876/task/34596/fd (opendir: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34597 safe_time ceph cwd unknown /proc/31876/task/34597/cwd (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34597 safe_time ceph rtd unknown /proc/31876/task/34597/root (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34597 safe_time ceph txt unknown /proc/31876/task/34597/exe (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34597 safe_time ceph NOFD /proc/31876/task/34597/fd (opendir: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34598 cmdfin ceph cwd unknown /proc/31876/task/34598/cwd (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34598 cmdfin ceph rtd unknown /proc/31876/task/34598/root (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34598 cmdfin ceph txt unknown /proc/31876/task/34598/exe (readlink: Permission denied) 2024-03-16T12:24:12.230 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34598 cmdfin ceph NOFD /proc/31876/task/34598/fd (opendir: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34602 io_contex ceph cwd unknown /proc/31876/task/34602/cwd (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34602 io_contex ceph rtd unknown /proc/31876/task/34602/root (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34602 io_contex ceph txt unknown /proc/31876/task/34602/exe (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34602 io_contex ceph NOFD /proc/31876/task/34602/fd (opendir: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34603 io_contex ceph cwd unknown /proc/31876/task/34603/cwd (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34603 io_contex ceph rtd unknown /proc/31876/task/34603/root (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34603 io_contex ceph txt unknown /proc/31876/task/34603/exe (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34603 io_contex ceph NOFD /proc/31876/task/34603/fd (opendir: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34604 ceph_time ceph cwd unknown /proc/31876/task/34604/cwd (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34604 ceph_time ceph rtd unknown /proc/31876/task/34604/root (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34604 ceph_time ceph txt unknown /proc/31876/task/34604/exe (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34604 ceph_time ceph NOFD /proc/31876/task/34604/fd (opendir: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34605 ms_dispat ceph cwd unknown /proc/31876/task/34605/cwd (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34605 ms_dispat ceph rtd unknown /proc/31876/task/34605/root (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34605 ms_dispat ceph txt unknown /proc/31876/task/34605/exe (readlink: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34605 ms_dispat ceph NOFD /proc/31876/task/34605/fd (opendir: Permission denied) 2024-03-16T12:24:12.231 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34606 ms_local ceph cwd unknown /proc/31876/task/34606/cwd (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34606 ms_local ceph rtd unknown /proc/31876/task/34606/root (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34606 ms_local ceph txt unknown /proc/31876/task/34606/exe (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34606 ms_local ceph NOFD /proc/31876/task/34606/fd (opendir: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34607 safe_time ceph cwd unknown /proc/31876/task/34607/cwd (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34607 safe_time ceph rtd unknown /proc/31876/task/34607/root (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34607 safe_time ceph txt unknown /proc/31876/task/34607/exe (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34607 safe_time ceph NOFD /proc/31876/task/34607/fd (opendir: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34608 safe_time ceph cwd unknown /proc/31876/task/34608/cwd (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34608 safe_time ceph rtd unknown /proc/31876/task/34608/root (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34608 safe_time ceph txt unknown /proc/31876/task/34608/exe (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34608 safe_time ceph NOFD /proc/31876/task/34608/fd (opendir: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34609 balancer ceph cwd unknown /proc/31876/task/34609/cwd (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34609 balancer ceph rtd unknown /proc/31876/task/34609/root (readlink: Permission denied) 2024-03-16T12:24:12.232 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34609 balancer ceph txt unknown /proc/31876/task/34609/exe (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34609 balancer ceph NOFD /proc/31876/task/34609/fd (opendir: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34610 m-fin-bal ceph cwd unknown /proc/31876/task/34610/cwd (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34610 m-fin-bal ceph rtd unknown /proc/31876/task/34610/root (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34610 m-fin-bal ceph txt unknown /proc/31876/task/34610/exe (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34610 m-fin-bal ceph NOFD /proc/31876/task/34610/fd (opendir: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34611 mgr-fin ceph cwd unknown /proc/31876/task/34611/cwd (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34611 mgr-fin ceph rtd unknown /proc/31876/task/34611/root (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34611 mgr-fin ceph txt unknown /proc/31876/task/34611/exe (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34611 mgr-fin ceph NOFD /proc/31876/task/34611/fd (opendir: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34612 mgr-fin ceph cwd unknown /proc/31876/task/34612/cwd (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34612 mgr-fin ceph rtd unknown /proc/31876/task/34612/root (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34612 mgr-fin ceph txt unknown /proc/31876/task/34612/exe (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34612 mgr-fin ceph NOFD /proc/31876/task/34612/fd (opendir: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34613 mgr-fin ceph cwd unknown /proc/31876/task/34613/cwd (readlink: Permission denied) 2024-03-16T12:24:12.233 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34613 mgr-fin ceph rtd unknown /proc/31876/task/34613/root (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34613 mgr-fin ceph txt unknown /proc/31876/task/34613/exe (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34613 mgr-fin ceph NOFD /proc/31876/task/34613/fd (opendir: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34614 mgr-fin ceph cwd unknown /proc/31876/task/34614/cwd (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34614 mgr-fin ceph rtd unknown /proc/31876/task/34614/root (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34614 mgr-fin ceph txt unknown /proc/31876/task/34614/exe (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34614 mgr-fin ceph NOFD /proc/31876/task/34614/fd (opendir: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34615 mgr-fin ceph cwd unknown /proc/31876/task/34615/cwd (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34615 mgr-fin ceph rtd unknown /proc/31876/task/34615/root (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34615 mgr-fin ceph txt unknown /proc/31876/task/34615/exe (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34615 mgr-fin ceph NOFD /proc/31876/task/34615/fd (opendir: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34616 mgr-fin ceph cwd unknown /proc/31876/task/34616/cwd (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34616 mgr-fin ceph rtd unknown /proc/31876/task/34616/root (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34616 mgr-fin ceph txt unknown /proc/31876/task/34616/exe (readlink: Permission denied) 2024-03-16T12:24:12.234 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34616 mgr-fin ceph NOFD /proc/31876/task/34616/fd (opendir: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34617 mgr-fin ceph cwd unknown /proc/31876/task/34617/cwd (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34617 mgr-fin ceph rtd unknown /proc/31876/task/34617/root (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34617 mgr-fin ceph txt unknown /proc/31876/task/34617/exe (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34617 mgr-fin ceph NOFD /proc/31876/task/34617/fd (opendir: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34618 mgr-fin ceph cwd unknown /proc/31876/task/34618/cwd (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34618 mgr-fin ceph rtd unknown /proc/31876/task/34618/root (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34618 mgr-fin ceph txt unknown /proc/31876/task/34618/exe (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34618 mgr-fin ceph NOFD /proc/31876/task/34618/fd (opendir: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34619 mgr-fin ceph cwd unknown /proc/31876/task/34619/cwd (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34619 mgr-fin ceph rtd unknown /proc/31876/task/34619/root (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34619 mgr-fin ceph txt unknown /proc/31876/task/34619/exe (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34619 mgr-fin ceph NOFD /proc/31876/task/34619/fd (opendir: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34620 mgr-fin ceph cwd unknown /proc/31876/task/34620/cwd (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34620 mgr-fin ceph rtd unknown /proc/31876/task/34620/root (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34620 mgr-fin ceph txt unknown /proc/31876/task/34620/exe (readlink: Permission denied) 2024-03-16T12:24:12.235 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34620 mgr-fin ceph NOFD /proc/31876/task/34620/fd (opendir: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34621 mgr-fin ceph cwd unknown /proc/31876/task/34621/cwd (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34621 mgr-fin ceph rtd unknown /proc/31876/task/34621/root (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34621 mgr-fin ceph txt unknown /proc/31876/task/34621/exe (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34621 mgr-fin ceph NOFD /proc/31876/task/34621/fd (opendir: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34622 mgr-fin ceph cwd unknown /proc/31876/task/34622/cwd (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34622 mgr-fin ceph rtd unknown /proc/31876/task/34622/root (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34622 mgr-fin ceph txt unknown /proc/31876/task/34622/exe (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34622 mgr-fin ceph NOFD /proc/31876/task/34622/fd (opendir: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34623 mgr-fin ceph cwd unknown /proc/31876/task/34623/cwd (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34623 mgr-fin ceph rtd unknown /proc/31876/task/34623/root (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34623 mgr-fin ceph txt unknown /proc/31876/task/34623/exe (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34623 mgr-fin ceph NOFD /proc/31876/task/34623/fd (opendir: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34624 mgr-fin ceph cwd unknown /proc/31876/task/34624/cwd (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34624 mgr-fin ceph rtd unknown /proc/31876/task/34624/root (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34624 mgr-fin ceph txt unknown /proc/31876/task/34624/exe (readlink: Permission denied) 2024-03-16T12:24:12.236 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34624 mgr-fin ceph NOFD /proc/31876/task/34624/fd (opendir: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34625 mgr-fin ceph cwd unknown /proc/31876/task/34625/cwd (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34625 mgr-fin ceph rtd unknown /proc/31876/task/34625/root (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34625 mgr-fin ceph txt unknown /proc/31876/task/34625/exe (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34625 mgr-fin ceph NOFD /proc/31876/task/34625/fd (opendir: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34626 mgr-fin ceph cwd unknown /proc/31876/task/34626/cwd (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34626 mgr-fin ceph rtd unknown /proc/31876/task/34626/root (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34626 mgr-fin ceph txt unknown /proc/31876/task/34626/exe (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34626 mgr-fin ceph NOFD /proc/31876/task/34626/fd (opendir: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34627 cephadm ceph cwd unknown /proc/31876/task/34627/cwd (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34627 cephadm ceph rtd unknown /proc/31876/task/34627/root (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34627 cephadm ceph txt unknown /proc/31876/task/34627/exe (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34627 cephadm ceph NOFD /proc/31876/task/34627/fd (opendir: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34628 mgr-fin ceph cwd unknown /proc/31876/task/34628/cwd (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34628 mgr-fin ceph rtd unknown /proc/31876/task/34628/root (readlink: Permission denied) 2024-03-16T12:24:12.237 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34628 mgr-fin ceph txt unknown /proc/31876/task/34628/exe (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34628 mgr-fin ceph NOFD /proc/31876/task/34628/fd (opendir: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34629 m-fin-cep ceph cwd unknown /proc/31876/task/34629/cwd (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34629 m-fin-cep ceph rtd unknown /proc/31876/task/34629/root (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34629 m-fin-cep ceph txt unknown /proc/31876/task/34629/exe (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34629 m-fin-cep ceph NOFD /proc/31876/task/34629/fd (opendir: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34630 mgr-fin ceph cwd unknown /proc/31876/task/34630/cwd (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34630 mgr-fin ceph rtd unknown /proc/31876/task/34630/root (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34630 mgr-fin ceph txt unknown /proc/31876/task/34630/exe (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34630 mgr-fin ceph NOFD /proc/31876/task/34630/fd (opendir: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34631 mgr-fin ceph cwd unknown /proc/31876/task/34631/cwd (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34631 mgr-fin ceph rtd unknown /proc/31876/task/34631/root (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34631 mgr-fin ceph txt unknown /proc/31876/task/34631/exe (readlink: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34631 mgr-fin ceph NOFD /proc/31876/task/34631/fd (opendir: Permission denied) 2024-03-16T12:24:12.238 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34632 crash ceph cwd unknown /proc/31876/task/34632/cwd (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34632 crash ceph rtd unknown /proc/31876/task/34632/root (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34632 crash ceph txt unknown /proc/31876/task/34632/exe (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34632 crash ceph NOFD /proc/31876/task/34632/fd (opendir: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34633 m-fin-cra ceph cwd unknown /proc/31876/task/34633/cwd (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34633 m-fin-cra ceph rtd unknown /proc/31876/task/34633/root (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34633 m-fin-cra ceph txt unknown /proc/31876/task/34633/exe (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34633 m-fin-cra ceph NOFD /proc/31876/task/34633/fd (opendir: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34634 mgr-fin ceph cwd unknown /proc/31876/task/34634/cwd (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34634 mgr-fin ceph rtd unknown /proc/31876/task/34634/root (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34634 mgr-fin ceph txt unknown /proc/31876/task/34634/exe (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34634 mgr-fin ceph NOFD /proc/31876/task/34634/fd (opendir: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34635 cephadm ceph cwd unknown /proc/31876/task/34635/cwd (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34635 cephadm ceph rtd unknown /proc/31876/task/34635/root (readlink: Permission denied) 2024-03-16T12:24:12.239 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34635 cephadm ceph txt unknown /proc/31876/task/34635/exe (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34635 cephadm ceph NOFD /proc/31876/task/34635/fd (opendir: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34636 mgr-fin ceph cwd unknown /proc/31876/task/34636/cwd (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34636 mgr-fin ceph rtd unknown /proc/31876/task/34636/root (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34636 mgr-fin ceph txt unknown /proc/31876/task/34636/exe (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34636 mgr-fin ceph NOFD /proc/31876/task/34636/fd (opendir: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34637 mgr-fin ceph cwd unknown /proc/31876/task/34637/cwd (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34637 mgr-fin ceph rtd unknown /proc/31876/task/34637/root (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34637 mgr-fin ceph txt unknown /proc/31876/task/34637/exe (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34637 mgr-fin ceph NOFD /proc/31876/task/34637/fd (opendir: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34638 dashboard ceph cwd unknown /proc/31876/task/34638/cwd (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34638 dashboard ceph rtd unknown /proc/31876/task/34638/root (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34638 dashboard ceph txt unknown /proc/31876/task/34638/exe (readlink: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34638 dashboard ceph NOFD /proc/31876/task/34638/fd (opendir: Permission denied) 2024-03-16T12:24:12.240 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34639 mgr-fin ceph cwd unknown /proc/31876/task/34639/cwd (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34639 mgr-fin ceph rtd unknown /proc/31876/task/34639/root (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34639 mgr-fin ceph txt unknown /proc/31876/task/34639/exe (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34639 mgr-fin ceph NOFD /proc/31876/task/34639/fd (opendir: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34640 m-fin-das ceph cwd unknown /proc/31876/task/34640/cwd (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34640 m-fin-das ceph rtd unknown /proc/31876/task/34640/root (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34640 m-fin-das ceph txt unknown /proc/31876/task/34640/exe (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34640 m-fin-das ceph NOFD /proc/31876/task/34640/fd (opendir: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34641 mgr-fin ceph cwd unknown /proc/31876/task/34641/cwd (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34641 mgr-fin ceph rtd unknown /proc/31876/task/34641/root (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34641 mgr-fin ceph txt unknown /proc/31876/task/34641/exe (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34641 mgr-fin ceph NOFD /proc/31876/task/34641/fd (opendir: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34642 devicehea ceph cwd unknown /proc/31876/task/34642/cwd (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34642 devicehea ceph rtd unknown /proc/31876/task/34642/root (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34642 devicehea ceph txt unknown /proc/31876/task/34642/exe (readlink: Permission denied) 2024-03-16T12:24:12.241 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34642 devicehea ceph NOFD /proc/31876/task/34642/fd (opendir: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34643 m-fin-dev ceph cwd unknown /proc/31876/task/34643/cwd (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34643 m-fin-dev ceph rtd unknown /proc/31876/task/34643/root (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34643 m-fin-dev ceph txt unknown /proc/31876/task/34643/exe (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34643 m-fin-dev ceph NOFD /proc/31876/task/34643/fd (opendir: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34644 mgr-fin ceph cwd unknown /proc/31876/task/34644/cwd (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34644 mgr-fin ceph rtd unknown /proc/31876/task/34644/root (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34644 mgr-fin ceph txt unknown /proc/31876/task/34644/exe (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34644 mgr-fin ceph NOFD /proc/31876/task/34644/fd (opendir: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34646 m-fin-ios ceph cwd unknown /proc/31876/task/34646/cwd (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34646 m-fin-ios ceph rtd unknown /proc/31876/task/34646/root (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34646 m-fin-ios ceph txt unknown /proc/31876/task/34646/exe (readlink: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34646 m-fin-ios ceph NOFD /proc/31876/task/34646/fd (opendir: Permission denied) 2024-03-16T12:24:12.242 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34647 mgr-fin ceph cwd unknown /proc/31876/task/34647/cwd (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34647 mgr-fin ceph rtd unknown /proc/31876/task/34647/root (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34647 mgr-fin ceph txt unknown /proc/31876/task/34647/exe (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34647 mgr-fin ceph NOFD /proc/31876/task/34647/fd (opendir: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34649 m-fin-nfs ceph cwd unknown /proc/31876/task/34649/cwd (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34649 m-fin-nfs ceph rtd unknown /proc/31876/task/34649/root (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34649 m-fin-nfs ceph txt unknown /proc/31876/task/34649/exe (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34649 m-fin-nfs ceph NOFD /proc/31876/task/34649/fd (opendir: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34650 mgr-fin ceph cwd unknown /proc/31876/task/34650/cwd (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34650 mgr-fin ceph rtd unknown /proc/31876/task/34650/root (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34650 mgr-fin ceph txt unknown /proc/31876/task/34650/exe (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34650 mgr-fin ceph NOFD /proc/31876/task/34650/fd (opendir: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34652 m-fin-orc ceph cwd unknown /proc/31876/task/34652/cwd (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34652 m-fin-orc ceph rtd unknown /proc/31876/task/34652/root (readlink: Permission denied) 2024-03-16T12:24:12.243 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34652 m-fin-orc ceph txt unknown /proc/31876/task/34652/exe (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34652 m-fin-orc ceph NOFD /proc/31876/task/34652/fd (opendir: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34653 pg_autosc ceph cwd unknown /proc/31876/task/34653/cwd (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34653 pg_autosc ceph rtd unknown /proc/31876/task/34653/root (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34653 pg_autosc ceph txt unknown /proc/31876/task/34653/exe (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34653 pg_autosc ceph NOFD /proc/31876/task/34653/fd (opendir: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34654 mgr-fin ceph cwd unknown /proc/31876/task/34654/cwd (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34654 mgr-fin ceph rtd unknown /proc/31876/task/34654/root (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34654 mgr-fin ceph txt unknown /proc/31876/task/34654/exe (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34654 mgr-fin ceph NOFD /proc/31876/task/34654/fd (opendir: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34655 m-fin-pg_ ceph cwd unknown /proc/31876/task/34655/cwd (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34655 m-fin-pg_ ceph rtd unknown /proc/31876/task/34655/root (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34655 m-fin-pg_ ceph txt unknown /proc/31876/task/34655/exe (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34655 m-fin-pg_ ceph NOFD /proc/31876/task/34655/fd (opendir: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34656 progress ceph cwd unknown /proc/31876/task/34656/cwd (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34656 progress ceph rtd unknown /proc/31876/task/34656/root (readlink: Permission denied) 2024-03-16T12:24:12.244 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34656 progress ceph txt unknown /proc/31876/task/34656/exe (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34656 progress ceph NOFD /proc/31876/task/34656/fd (opendir: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34657 m-fin-pro ceph cwd unknown /proc/31876/task/34657/cwd (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34657 m-fin-pro ceph rtd unknown /proc/31876/task/34657/root (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34657 m-fin-pro ceph txt unknown /proc/31876/task/34657/exe (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34657 m-fin-pro ceph NOFD /proc/31876/task/34657/fd (opendir: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34661 io_contex ceph cwd unknown /proc/31876/task/34661/cwd (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34661 io_contex ceph rtd unknown /proc/31876/task/34661/root (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34661 io_contex ceph txt unknown /proc/31876/task/34661/exe (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34661 io_contex ceph NOFD /proc/31876/task/34661/fd (opendir: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34662 io_contex ceph cwd unknown /proc/31876/task/34662/cwd (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34662 io_contex ceph rtd unknown /proc/31876/task/34662/root (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34662 io_contex ceph txt unknown /proc/31876/task/34662/exe (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34662 io_contex ceph NOFD /proc/31876/task/34662/fd (opendir: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34663 ceph_time ceph cwd unknown /proc/31876/task/34663/cwd (readlink: Permission denied) 2024-03-16T12:24:12.245 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34663 ceph_time ceph rtd unknown /proc/31876/task/34663/root (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34663 ceph_time ceph txt unknown /proc/31876/task/34663/exe (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34663 ceph_time ceph NOFD /proc/31876/task/34663/fd (opendir: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34664 ms_dispat ceph cwd unknown /proc/31876/task/34664/cwd (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34664 ms_dispat ceph rtd unknown /proc/31876/task/34664/root (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34664 ms_dispat ceph txt unknown /proc/31876/task/34664/exe (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34664 ms_dispat ceph NOFD /proc/31876/task/34664/fd (opendir: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34665 ms_local ceph cwd unknown /proc/31876/task/34665/cwd (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34665 ms_local ceph rtd unknown /proc/31876/task/34665/root (readlink: Permission denied) 2024-03-16T12:24:12.246 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34665 ms_local ceph txt unknown /proc/31876/task/34665/exe (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34665 ms_local ceph NOFD /proc/31876/task/34665/fd (opendir: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34666 safe_time ceph cwd unknown /proc/31876/task/34666/cwd (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34666 safe_time ceph rtd unknown /proc/31876/task/34666/root (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34666 safe_time ceph txt unknown /proc/31876/task/34666/exe (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34666 safe_time ceph NOFD /proc/31876/task/34666/fd (opendir: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34667 safe_time ceph cwd unknown /proc/31876/task/34667/cwd (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34667 safe_time ceph rtd unknown /proc/31876/task/34667/root (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34667 safe_time ceph txt unknown /proc/31876/task/34667/exe (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34667 safe_time ceph NOFD /proc/31876/task/34667/fd (opendir: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34668 mgr-fin ceph cwd unknown /proc/31876/task/34668/cwd (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34668 mgr-fin ceph rtd unknown /proc/31876/task/34668/root (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34668 mgr-fin ceph txt unknown /proc/31876/task/34668/exe (readlink: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34668 mgr-fin ceph NOFD /proc/31876/task/34668/fd (opendir: Permission denied) 2024-03-16T12:24:12.247 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34670 m-fin-rbd ceph cwd unknown /proc/31876/task/34670/cwd (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34670 m-fin-rbd ceph rtd unknown /proc/31876/task/34670/root (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34670 m-fin-rbd ceph txt unknown /proc/31876/task/34670/exe (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34670 m-fin-rbd ceph NOFD /proc/31876/task/34670/fd (opendir: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34671 restful ceph cwd unknown /proc/31876/task/34671/cwd (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34671 restful ceph rtd unknown /proc/31876/task/34671/root (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34671 restful ceph txt unknown /proc/31876/task/34671/exe (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34671 restful ceph NOFD /proc/31876/task/34671/fd (opendir: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34672 m-fin-res ceph cwd unknown /proc/31876/task/34672/cwd (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34672 m-fin-res ceph rtd unknown /proc/31876/task/34672/root (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34672 m-fin-res ceph txt unknown /proc/31876/task/34672/exe (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34672 m-fin-res ceph NOFD /proc/31876/task/34672/fd (opendir: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34674 m-fin-sta ceph cwd unknown /proc/31876/task/34674/cwd (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34674 m-fin-sta ceph rtd unknown /proc/31876/task/34674/root (readlink: Permission denied) 2024-03-16T12:24:12.248 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34674 m-fin-sta ceph txt unknown /proc/31876/task/34674/exe (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34674 m-fin-sta ceph NOFD /proc/31876/task/34674/fd (opendir: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34675 telemetry ceph cwd unknown /proc/31876/task/34675/cwd (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34675 telemetry ceph rtd unknown /proc/31876/task/34675/root (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34675 telemetry ceph txt unknown /proc/31876/task/34675/exe (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34675 telemetry ceph NOFD /proc/31876/task/34675/fd (opendir: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34676 m-fin-tel ceph cwd unknown /proc/31876/task/34676/cwd (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34676 m-fin-tel ceph rtd unknown /proc/31876/task/34676/root (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34676 m-fin-tel ceph txt unknown /proc/31876/task/34676/exe (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34676 m-fin-tel ceph NOFD /proc/31876/task/34676/fd (opendir: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34677 mgr-fin ceph cwd unknown /proc/31876/task/34677/cwd (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34677 mgr-fin ceph rtd unknown /proc/31876/task/34677/root (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34677 mgr-fin ceph txt unknown /proc/31876/task/34677/exe (readlink: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34677 mgr-fin ceph NOFD /proc/31876/task/34677/fd (opendir: Permission denied) 2024-03-16T12:24:12.249 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34678 mgr-fin ceph cwd unknown /proc/31876/task/34678/cwd (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34678 mgr-fin ceph rtd unknown /proc/31876/task/34678/root (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34678 mgr-fin ceph txt unknown /proc/31876/task/34678/exe (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34678 mgr-fin ceph NOFD /proc/31876/task/34678/fd (opendir: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34680 mgr-fin ceph cwd unknown /proc/31876/task/34680/cwd (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34680 mgr-fin ceph rtd unknown /proc/31876/task/34680/root (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34680 mgr-fin ceph txt unknown /proc/31876/task/34680/exe (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34680 mgr-fin ceph NOFD /proc/31876/task/34680/fd (opendir: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34683 mgr-fin ceph cwd unknown /proc/31876/task/34683/cwd (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34683 mgr-fin ceph rtd unknown /proc/31876/task/34683/root (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34683 mgr-fin ceph txt unknown /proc/31876/task/34683/exe (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34683 mgr-fin ceph NOFD /proc/31876/task/34683/fd (opendir: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34685 mgr-fin ceph cwd unknown /proc/31876/task/34685/cwd (readlink: Permission denied) 2024-03-16T12:24:12.250 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34685 mgr-fin ceph rtd unknown /proc/31876/task/34685/root (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34685 mgr-fin ceph txt unknown /proc/31876/task/34685/exe (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34685 mgr-fin ceph NOFD /proc/31876/task/34685/fd (opendir: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34686 io_contex ceph cwd unknown /proc/31876/task/34686/cwd (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34686 io_contex ceph rtd unknown /proc/31876/task/34686/root (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34686 io_contex ceph txt unknown /proc/31876/task/34686/exe (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34686 io_contex ceph NOFD /proc/31876/task/34686/fd (opendir: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34687 io_contex ceph cwd unknown /proc/31876/task/34687/cwd (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34687 io_contex ceph rtd unknown /proc/31876/task/34687/root (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34687 io_contex ceph txt unknown /proc/31876/task/34687/exe (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34687 io_contex ceph NOFD /proc/31876/task/34687/fd (opendir: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34688 ceph_time ceph cwd unknown /proc/31876/task/34688/cwd (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34688 ceph_time ceph rtd unknown /proc/31876/task/34688/root (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34688 ceph_time ceph txt unknown /proc/31876/task/34688/exe (readlink: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34688 ceph_time ceph NOFD /proc/31876/task/34688/fd (opendir: Permission denied) 2024-03-16T12:24:12.251 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34689 ms_dispat ceph cwd unknown /proc/31876/task/34689/cwd (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34689 ms_dispat ceph rtd unknown /proc/31876/task/34689/root (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34689 ms_dispat ceph txt unknown /proc/31876/task/34689/exe (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34689 ms_dispat ceph NOFD /proc/31876/task/34689/fd (opendir: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34690 ms_local ceph cwd unknown /proc/31876/task/34690/cwd (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34690 ms_local ceph rtd unknown /proc/31876/task/34690/root (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34690 ms_local ceph txt unknown /proc/31876/task/34690/exe (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34690 ms_local ceph NOFD /proc/31876/task/34690/fd (opendir: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34691 safe_time ceph cwd unknown /proc/31876/task/34691/cwd (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34691 safe_time ceph rtd unknown /proc/31876/task/34691/root (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34691 safe_time ceph txt unknown /proc/31876/task/34691/exe (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34691 safe_time ceph NOFD /proc/31876/task/34691/fd (opendir: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34692 safe_time ceph cwd unknown /proc/31876/task/34692/cwd (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34692 safe_time ceph rtd unknown /proc/31876/task/34692/root (readlink: Permission denied) 2024-03-16T12:24:12.252 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34692 safe_time ceph txt unknown /proc/31876/task/34692/exe (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34692 safe_time ceph NOFD /proc/31876/task/34692/fd (opendir: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34693 mgr-fin ceph cwd unknown /proc/31876/task/34693/cwd (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34693 mgr-fin ceph rtd unknown /proc/31876/task/34693/root (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34693 mgr-fin ceph txt unknown /proc/31876/task/34693/exe (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34693 mgr-fin ceph NOFD /proc/31876/task/34693/fd (opendir: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34694 mgr-fin ceph cwd unknown /proc/31876/task/34694/cwd (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34694 mgr-fin ceph rtd unknown /proc/31876/task/34694/root (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34694 mgr-fin ceph txt unknown /proc/31876/task/34694/exe (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34694 mgr-fin ceph NOFD /proc/31876/task/34694/fd (opendir: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34695 mgr-fin ceph cwd unknown /proc/31876/task/34695/cwd (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34695 mgr-fin ceph rtd unknown /proc/31876/task/34695/root (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34695 mgr-fin ceph txt unknown /proc/31876/task/34695/exe (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34695 mgr-fin ceph NOFD /proc/31876/task/34695/fd (opendir: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34696 mgr-fin ceph cwd unknown /proc/31876/task/34696/cwd (readlink: Permission denied) 2024-03-16T12:24:12.253 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34696 mgr-fin ceph rtd unknown /proc/31876/task/34696/root (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34696 mgr-fin ceph txt unknown /proc/31876/task/34696/exe (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34696 mgr-fin ceph NOFD /proc/31876/task/34696/fd (opendir: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34697 mgr-fin ceph cwd unknown /proc/31876/task/34697/cwd (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34697 mgr-fin ceph rtd unknown /proc/31876/task/34697/root (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34697 mgr-fin ceph txt unknown /proc/31876/task/34697/exe (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34697 mgr-fin ceph NOFD /proc/31876/task/34697/fd (opendir: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34698 mgr-fin ceph cwd unknown /proc/31876/task/34698/cwd (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34698 mgr-fin ceph rtd unknown /proc/31876/task/34698/root (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34698 mgr-fin ceph txt unknown /proc/31876/task/34698/exe (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34698 mgr-fin ceph NOFD /proc/31876/task/34698/fd (opendir: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34699 mgr-fin ceph cwd unknown /proc/31876/task/34699/cwd (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34699 mgr-fin ceph rtd unknown /proc/31876/task/34699/root (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34699 mgr-fin ceph txt unknown /proc/31876/task/34699/exe (readlink: Permission denied) 2024-03-16T12:24:12.254 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34699 mgr-fin ceph NOFD /proc/31876/task/34699/fd (opendir: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34700 mgr-fin ceph cwd unknown /proc/31876/task/34700/cwd (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34700 mgr-fin ceph rtd unknown /proc/31876/task/34700/root (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34700 mgr-fin ceph txt unknown /proc/31876/task/34700/exe (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34700 mgr-fin ceph NOFD /proc/31876/task/34700/fd (opendir: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34701 mgr-fin ceph cwd unknown /proc/31876/task/34701/cwd (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34701 mgr-fin ceph rtd unknown /proc/31876/task/34701/root (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34701 mgr-fin ceph txt unknown /proc/31876/task/34701/exe (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34701 mgr-fin ceph NOFD /proc/31876/task/34701/fd (opendir: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34702 mgr-fin ceph cwd unknown /proc/31876/task/34702/cwd (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34702 mgr-fin ceph rtd unknown /proc/31876/task/34702/root (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34702 mgr-fin ceph txt unknown /proc/31876/task/34702/exe (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34702 mgr-fin ceph NOFD /proc/31876/task/34702/fd (opendir: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34703 mgr-fin ceph cwd unknown /proc/31876/task/34703/cwd (readlink: Permission denied) 2024-03-16T12:24:12.255 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34703 mgr-fin ceph rtd unknown /proc/31876/task/34703/root (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34703 mgr-fin ceph txt unknown /proc/31876/task/34703/exe (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34703 mgr-fin ceph NOFD /proc/31876/task/34703/fd (opendir: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34704 mgr-fin ceph cwd unknown /proc/31876/task/34704/cwd (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34704 mgr-fin ceph rtd unknown /proc/31876/task/34704/root (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34704 mgr-fin ceph txt unknown /proc/31876/task/34704/exe (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34704 mgr-fin ceph NOFD /proc/31876/task/34704/fd (opendir: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34706 m-fin-vol ceph cwd unknown /proc/31876/task/34706/cwd (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34706 m-fin-vol ceph rtd unknown /proc/31876/task/34706/root (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34706 m-fin-vol ceph txt unknown /proc/31876/task/34706/exe (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34706 m-fin-vol ceph NOFD /proc/31876/task/34706/fd (opendir: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34724 mgr-fin ceph cwd unknown /proc/31876/task/34724/cwd (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34724 mgr-fin ceph rtd unknown /proc/31876/task/34724/root (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34724 mgr-fin ceph txt unknown /proc/31876/task/34724/exe (readlink: Permission denied) 2024-03-16T12:24:12.256 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34724 mgr-fin ceph NOFD /proc/31876/task/34724/fd (opendir: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34725 mgr-fin ceph cwd unknown /proc/31876/task/34725/cwd (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34725 mgr-fin ceph rtd unknown /proc/31876/task/34725/root (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34725 mgr-fin ceph txt unknown /proc/31876/task/34725/exe (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34725 mgr-fin ceph NOFD /proc/31876/task/34725/fd (opendir: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34726 mgr-fin ceph cwd unknown /proc/31876/task/34726/cwd (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34726 mgr-fin ceph rtd unknown /proc/31876/task/34726/root (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34726 mgr-fin ceph txt unknown /proc/31876/task/34726/exe (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34726 mgr-fin ceph NOFD /proc/31876/task/34726/fd (opendir: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34727 mgr-fin ceph cwd unknown /proc/31876/task/34727/cwd (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34727 mgr-fin ceph rtd unknown /proc/31876/task/34727/root (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34727 mgr-fin ceph txt unknown /proc/31876/task/34727/exe (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34727 mgr-fin ceph NOFD /proc/31876/task/34727/fd (opendir: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34728 mgr-fin ceph cwd unknown /proc/31876/task/34728/cwd (readlink: Permission denied) 2024-03-16T12:24:12.257 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34728 mgr-fin ceph rtd unknown /proc/31876/task/34728/root (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34728 mgr-fin ceph txt unknown /proc/31876/task/34728/exe (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34728 mgr-fin ceph NOFD /proc/31876/task/34728/fd (opendir: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34729 mgr-fin ceph cwd unknown /proc/31876/task/34729/cwd (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34729 mgr-fin ceph rtd unknown /proc/31876/task/34729/root (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34729 mgr-fin ceph txt unknown /proc/31876/task/34729/exe (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34729 mgr-fin ceph NOFD /proc/31876/task/34729/fd (opendir: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34730 mgr-fin ceph cwd unknown /proc/31876/task/34730/cwd (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34730 mgr-fin ceph rtd unknown /proc/31876/task/34730/root (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34730 mgr-fin ceph txt unknown /proc/31876/task/34730/exe (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34730 mgr-fin ceph NOFD /proc/31876/task/34730/fd (opendir: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34731 mgr-fin ceph cwd unknown /proc/31876/task/34731/cwd (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34731 mgr-fin ceph rtd unknown /proc/31876/task/34731/root (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34731 mgr-fin ceph txt unknown /proc/31876/task/34731/exe (readlink: Permission denied) 2024-03-16T12:24:12.258 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34731 mgr-fin ceph NOFD /proc/31876/task/34731/fd (opendir: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34732 mgr-fin ceph cwd unknown /proc/31876/task/34732/cwd (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34732 mgr-fin ceph rtd unknown /proc/31876/task/34732/root (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34732 mgr-fin ceph txt unknown /proc/31876/task/34732/exe (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34732 mgr-fin ceph NOFD /proc/31876/task/34732/fd (opendir: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34733 mgr-fin ceph cwd unknown /proc/31876/task/34733/cwd (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34733 mgr-fin ceph rtd unknown /proc/31876/task/34733/root (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34733 mgr-fin ceph txt unknown /proc/31876/task/34733/exe (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34733 mgr-fin ceph NOFD /proc/31876/task/34733/fd (opendir: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34734 mgr-fin ceph cwd unknown /proc/31876/task/34734/cwd (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34734 mgr-fin ceph rtd unknown /proc/31876/task/34734/root (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34734 mgr-fin ceph txt unknown /proc/31876/task/34734/exe (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34734 mgr-fin ceph NOFD /proc/31876/task/34734/fd (opendir: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34787 mgr-fin ceph cwd unknown /proc/31876/task/34787/cwd (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34787 mgr-fin ceph rtd unknown /proc/31876/task/34787/root (readlink: Permission denied) 2024-03-16T12:24:12.259 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34787 mgr-fin ceph txt unknown /proc/31876/task/34787/exe (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34787 mgr-fin ceph NOFD /proc/31876/task/34787/fd (opendir: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34789 mgr-fin ceph cwd unknown /proc/31876/task/34789/cwd (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34789 mgr-fin ceph rtd unknown /proc/31876/task/34789/root (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34789 mgr-fin ceph txt unknown /proc/31876/task/34789/exe (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34789 mgr-fin ceph NOFD /proc/31876/task/34789/fd (opendir: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34790 mgr-fin ceph cwd unknown /proc/31876/task/34790/cwd (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34790 mgr-fin ceph rtd unknown /proc/31876/task/34790/root (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34790 mgr-fin ceph txt unknown /proc/31876/task/34790/exe (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34790 mgr-fin ceph NOFD /proc/31876/task/34790/fd (opendir: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34791 mgr-fin ceph cwd unknown /proc/31876/task/34791/cwd (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34791 mgr-fin ceph rtd unknown /proc/31876/task/34791/root (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34791 mgr-fin ceph txt unknown /proc/31876/task/34791/exe (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34791 mgr-fin ceph NOFD /proc/31876/task/34791/fd (opendir: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34792 mgr-fin ceph cwd unknown /proc/31876/task/34792/cwd (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34792 mgr-fin ceph rtd unknown /proc/31876/task/34792/root (readlink: Permission denied) 2024-03-16T12:24:12.260 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34792 mgr-fin ceph txt unknown /proc/31876/task/34792/exe (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34792 mgr-fin ceph NOFD /proc/31876/task/34792/fd (opendir: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34793 mgr-fin ceph cwd unknown /proc/31876/task/34793/cwd (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34793 mgr-fin ceph rtd unknown /proc/31876/task/34793/root (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34793 mgr-fin ceph txt unknown /proc/31876/task/34793/exe (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34793 mgr-fin ceph NOFD /proc/31876/task/34793/fd (opendir: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34794 mgr-fin ceph cwd unknown /proc/31876/task/34794/cwd (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34794 mgr-fin ceph rtd unknown /proc/31876/task/34794/root (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34794 mgr-fin ceph txt unknown /proc/31876/task/34794/exe (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34794 mgr-fin ceph NOFD /proc/31876/task/34794/fd (opendir: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34795 mgr-fin ceph cwd unknown /proc/31876/task/34795/cwd (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34795 mgr-fin ceph rtd unknown /proc/31876/task/34795/root (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34795 mgr-fin ceph txt unknown /proc/31876/task/34795/exe (readlink: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34795 mgr-fin ceph NOFD /proc/31876/task/34795/fd (opendir: Permission denied) 2024-03-16T12:24:12.261 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34796 mgr-fin ceph cwd unknown /proc/31876/task/34796/cwd (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34796 mgr-fin ceph rtd unknown /proc/31876/task/34796/root (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34796 mgr-fin ceph txt unknown /proc/31876/task/34796/exe (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34796 mgr-fin ceph NOFD /proc/31876/task/34796/fd (opendir: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34797 mgr-fin ceph cwd unknown /proc/31876/task/34797/cwd (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34797 mgr-fin ceph rtd unknown /proc/31876/task/34797/root (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34797 mgr-fin ceph txt unknown /proc/31876/task/34797/exe (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34797 mgr-fin ceph NOFD /proc/31876/task/34797/fd (opendir: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34798 mgr-fin ceph cwd unknown /proc/31876/task/34798/cwd (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34798 mgr-fin ceph rtd unknown /proc/31876/task/34798/root (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34798 mgr-fin ceph txt unknown /proc/31876/task/34798/exe (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 34798 mgr-fin ceph NOFD /proc/31876/task/34798/fd (opendir: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35320 dashboard ceph cwd unknown /proc/31876/task/35320/cwd (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35320 dashboard ceph rtd unknown /proc/31876/task/35320/root (readlink: Permission denied) 2024-03-16T12:24:12.262 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35320 dashboard ceph txt unknown /proc/31876/task/35320/exe (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35320 dashboard ceph NOFD /proc/31876/task/35320/fd (opendir: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35321 dashboard ceph cwd unknown /proc/31876/task/35321/cwd (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35321 dashboard ceph rtd unknown /proc/31876/task/35321/root (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35321 dashboard ceph txt unknown /proc/31876/task/35321/exe (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35321 dashboard ceph NOFD /proc/31876/task/35321/fd (opendir: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35322 dashboard ceph cwd unknown /proc/31876/task/35322/cwd (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35322 dashboard ceph rtd unknown /proc/31876/task/35322/root (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35322 dashboard ceph txt unknown /proc/31876/task/35322/exe (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35322 dashboard ceph NOFD /proc/31876/task/35322/fd (opendir: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35323 dashboard ceph cwd unknown /proc/31876/task/35323/cwd (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35323 dashboard ceph rtd unknown /proc/31876/task/35323/root (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35323 dashboard ceph txt unknown /proc/31876/task/35323/exe (readlink: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35323 dashboard ceph NOFD /proc/31876/task/35323/fd (opendir: Permission denied) 2024-03-16T12:24:12.263 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35324 dashboard ceph cwd unknown /proc/31876/task/35324/cwd (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35324 dashboard ceph rtd unknown /proc/31876/task/35324/root (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35324 dashboard ceph txt unknown /proc/31876/task/35324/exe (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35324 dashboard ceph NOFD /proc/31876/task/35324/fd (opendir: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35325 dashboard ceph cwd unknown /proc/31876/task/35325/cwd (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35325 dashboard ceph rtd unknown /proc/31876/task/35325/root (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35325 dashboard ceph txt unknown /proc/31876/task/35325/exe (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35325 dashboard ceph NOFD /proc/31876/task/35325/fd (opendir: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35326 dashboard ceph cwd unknown /proc/31876/task/35326/cwd (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35326 dashboard ceph rtd unknown /proc/31876/task/35326/root (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35326 dashboard ceph txt unknown /proc/31876/task/35326/exe (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35326 dashboard ceph NOFD /proc/31876/task/35326/fd (opendir: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35327 dashboard ceph cwd unknown /proc/31876/task/35327/cwd (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35327 dashboard ceph rtd unknown /proc/31876/task/35327/root (readlink: Permission denied) 2024-03-16T12:24:12.264 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35327 dashboard ceph txt unknown /proc/31876/task/35327/exe (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35327 dashboard ceph NOFD /proc/31876/task/35327/fd (opendir: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35328 dashboard ceph cwd unknown /proc/31876/task/35328/cwd (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35328 dashboard ceph rtd unknown /proc/31876/task/35328/root (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35328 dashboard ceph txt unknown /proc/31876/task/35328/exe (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35328 dashboard ceph NOFD /proc/31876/task/35328/fd (opendir: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35329 dashboard ceph cwd unknown /proc/31876/task/35329/cwd (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35329 dashboard ceph rtd unknown /proc/31876/task/35329/root (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35329 dashboard ceph txt unknown /proc/31876/task/35329/exe (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35329 dashboard ceph NOFD /proc/31876/task/35329/fd (opendir: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35330 dashboard ceph cwd unknown /proc/31876/task/35330/cwd (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35330 dashboard ceph rtd unknown /proc/31876/task/35330/root (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35330 dashboard ceph txt unknown /proc/31876/task/35330/exe (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35330 dashboard ceph NOFD /proc/31876/task/35330/fd (opendir: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35333 dashboard ceph cwd unknown /proc/31876/task/35333/cwd (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35333 dashboard ceph rtd unknown /proc/31876/task/35333/root (readlink: Permission denied) 2024-03-16T12:24:12.265 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35333 dashboard ceph txt unknown /proc/31876/task/35333/exe (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 35333 dashboard ceph NOFD /proc/31876/task/35333/fd (opendir: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41904 cephadm ceph cwd unknown /proc/31876/task/41904/cwd (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41904 cephadm ceph rtd unknown /proc/31876/task/41904/root (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41904 cephadm ceph txt unknown /proc/31876/task/41904/exe (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41904 cephadm ceph NOFD /proc/31876/task/41904/fd (opendir: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41905 cephadm ceph cwd unknown /proc/31876/task/41905/cwd (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41905 cephadm ceph rtd unknown /proc/31876/task/41905/root (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41905 cephadm ceph txt unknown /proc/31876/task/41905/exe (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41905 cephadm ceph NOFD /proc/31876/task/41905/fd (opendir: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41906 cephadm ceph cwd unknown /proc/31876/task/41906/cwd (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41906 cephadm ceph rtd unknown /proc/31876/task/41906/root (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41906 cephadm ceph txt unknown /proc/31876/task/41906/exe (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41906 cephadm ceph NOFD /proc/31876/task/41906/fd (opendir: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41907 cephadm ceph cwd unknown /proc/31876/task/41907/cwd (readlink: Permission denied) 2024-03-16T12:24:12.266 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41907 cephadm ceph rtd unknown /proc/31876/task/41907/root (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41907 cephadm ceph txt unknown /proc/31876/task/41907/exe (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41907 cephadm ceph NOFD /proc/31876/task/41907/fd (opendir: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41908 cephadm ceph cwd unknown /proc/31876/task/41908/cwd (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41908 cephadm ceph rtd unknown /proc/31876/task/41908/root (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41908 cephadm ceph txt unknown /proc/31876/task/41908/exe (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41908 cephadm ceph NOFD /proc/31876/task/41908/fd (opendir: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41909 cephadm ceph cwd unknown /proc/31876/task/41909/cwd (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41909 cephadm ceph rtd unknown /proc/31876/task/41909/root (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41909 cephadm ceph txt unknown /proc/31876/task/41909/exe (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41909 cephadm ceph NOFD /proc/31876/task/41909/fd (opendir: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41910 cephadm ceph cwd unknown /proc/31876/task/41910/cwd (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41910 cephadm ceph rtd unknown /proc/31876/task/41910/root (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41910 cephadm ceph txt unknown /proc/31876/task/41910/exe (readlink: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41910 cephadm ceph NOFD /proc/31876/task/41910/fd (opendir: Permission denied) 2024-03-16T12:24:12.267 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41911 cephadm ceph cwd unknown /proc/31876/task/41911/cwd (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41911 cephadm ceph rtd unknown /proc/31876/task/41911/root (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41911 cephadm ceph txt unknown /proc/31876/task/41911/exe (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41911 cephadm ceph NOFD /proc/31876/task/41911/fd (opendir: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41912 cephadm ceph cwd unknown /proc/31876/task/41912/cwd (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41912 cephadm ceph rtd unknown /proc/31876/task/41912/root (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41912 cephadm ceph txt unknown /proc/31876/task/41912/exe (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41912 cephadm ceph NOFD /proc/31876/task/41912/fd (opendir: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41913 cephadm ceph cwd unknown /proc/31876/task/41913/cwd (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41913 cephadm ceph rtd unknown /proc/31876/task/41913/root (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41913 cephadm ceph txt unknown /proc/31876/task/41913/exe (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 41913 cephadm ceph NOFD /proc/31876/task/41913/fd (opendir: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42265 cephadm ceph cwd unknown /proc/31876/task/42265/cwd (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42265 cephadm ceph rtd unknown /proc/31876/task/42265/root (readlink: Permission denied) 2024-03-16T12:24:12.268 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42265 cephadm ceph txt unknown /proc/31876/task/42265/exe (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42265 cephadm ceph NOFD /proc/31876/task/42265/fd (opendir: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42266 cephadm ceph cwd unknown /proc/31876/task/42266/cwd (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42266 cephadm ceph rtd unknown /proc/31876/task/42266/root (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42266 cephadm ceph txt unknown /proc/31876/task/42266/exe (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42266 cephadm ceph NOFD /proc/31876/task/42266/fd (opendir: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42267 cephadm ceph cwd unknown /proc/31876/task/42267/cwd (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42267 cephadm ceph rtd unknown /proc/31876/task/42267/root (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42267 cephadm ceph txt unknown /proc/31876/task/42267/exe (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42267 cephadm ceph NOFD /proc/31876/task/42267/fd (opendir: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42268 cephadm ceph cwd unknown /proc/31876/task/42268/cwd (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42268 cephadm ceph rtd unknown /proc/31876/task/42268/root (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42268 cephadm ceph txt unknown /proc/31876/task/42268/exe (readlink: Permission denied) 2024-03-16T12:24:12.269 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42268 cephadm ceph NOFD /proc/31876/task/42268/fd (opendir: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42269 cephadm ceph cwd unknown /proc/31876/task/42269/cwd (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42269 cephadm ceph rtd unknown /proc/31876/task/42269/root (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42269 cephadm ceph txt unknown /proc/31876/task/42269/exe (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42269 cephadm ceph NOFD /proc/31876/task/42269/fd (opendir: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42270 cephadm ceph cwd unknown /proc/31876/task/42270/cwd (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42270 cephadm ceph rtd unknown /proc/31876/task/42270/root (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42270 cephadm ceph txt unknown /proc/31876/task/42270/exe (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42270 cephadm ceph NOFD /proc/31876/task/42270/fd (opendir: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42271 cephadm ceph cwd unknown /proc/31876/task/42271/cwd (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42271 cephadm ceph rtd unknown /proc/31876/task/42271/root (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42271 cephadm ceph txt unknown /proc/31876/task/42271/exe (readlink: Permission denied) 2024-03-16T12:24:12.270 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42271 cephadm ceph NOFD /proc/31876/task/42271/fd (opendir: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42272 cephadm ceph cwd unknown /proc/31876/task/42272/cwd (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42272 cephadm ceph rtd unknown /proc/31876/task/42272/root (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42272 cephadm ceph txt unknown /proc/31876/task/42272/exe (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42272 cephadm ceph NOFD /proc/31876/task/42272/fd (opendir: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42273 cephadm ceph cwd unknown /proc/31876/task/42273/cwd (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42273 cephadm ceph rtd unknown /proc/31876/task/42273/root (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42273 cephadm ceph txt unknown /proc/31876/task/42273/exe (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42273 cephadm ceph NOFD /proc/31876/task/42273/fd (opendir: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42274 cephadm ceph cwd unknown /proc/31876/task/42274/cwd (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42274 cephadm ceph rtd unknown /proc/31876/task/42274/root (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42274 cephadm ceph txt unknown /proc/31876/task/42274/exe (readlink: Permission denied) 2024-03-16T12:24:12.271 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 42274 cephadm ceph NOFD /proc/31876/task/42274/fd (opendir: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51050 devicehea ceph cwd unknown /proc/31876/task/51050/cwd (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51050 devicehea ceph rtd unknown /proc/31876/task/51050/root (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51050 devicehea ceph txt unknown /proc/31876/task/51050/exe (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51050 devicehea ceph NOFD /proc/31876/task/51050/fd (opendir: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51071 io_contex ceph cwd unknown /proc/31876/task/51071/cwd (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51071 io_contex ceph rtd unknown /proc/31876/task/51071/root (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51071 io_contex ceph txt unknown /proc/31876/task/51071/exe (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51071 io_contex ceph NOFD /proc/31876/task/51071/fd (opendir: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51072 io_contex ceph cwd unknown /proc/31876/task/51072/cwd (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51072 io_contex ceph rtd unknown /proc/31876/task/51072/root (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51072 io_contex ceph txt unknown /proc/31876/task/51072/exe (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51072 io_contex ceph NOFD /proc/31876/task/51072/fd (opendir: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51073 ceph_time ceph cwd unknown /proc/31876/task/51073/cwd (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51073 ceph_time ceph rtd unknown /proc/31876/task/51073/root (readlink: Permission denied) 2024-03-16T12:24:12.272 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51073 ceph_time ceph txt unknown /proc/31876/task/51073/exe (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51073 ceph_time ceph NOFD /proc/31876/task/51073/fd (opendir: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51074 ms_dispat ceph cwd unknown /proc/31876/task/51074/cwd (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51074 ms_dispat ceph rtd unknown /proc/31876/task/51074/root (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51074 ms_dispat ceph txt unknown /proc/31876/task/51074/exe (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51074 ms_dispat ceph NOFD /proc/31876/task/51074/fd (opendir: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51075 ms_local ceph cwd unknown /proc/31876/task/51075/cwd (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51075 ms_local ceph rtd unknown /proc/31876/task/51075/root (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51075 ms_local ceph txt unknown /proc/31876/task/51075/exe (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51075 ms_local ceph NOFD /proc/31876/task/51075/fd (opendir: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51076 safe_time ceph cwd unknown /proc/31876/task/51076/cwd (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51076 safe_time ceph rtd unknown /proc/31876/task/51076/root (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51076 safe_time ceph txt unknown /proc/31876/task/51076/exe (readlink: Permission denied) 2024-03-16T12:24:12.273 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51076 safe_time ceph NOFD /proc/31876/task/51076/fd (opendir: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51077 safe_time ceph cwd unknown /proc/31876/task/51077/cwd (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51077 safe_time ceph rtd unknown /proc/31876/task/51077/root (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51077 safe_time ceph txt unknown /proc/31876/task/51077/exe (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mgr 31876 51077 safe_time ceph NOFD /proc/31876/task/51077/fd (opendir: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 33016 root cwd unknown /proc/33016/cwd (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 33016 root rtd unknown /proc/33016/root (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 33016 root txt unknown /proc/33016/exe (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:systemd 33016 root NOFD /proc/33016/fd (opendir: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 33019 root cwd unknown /proc/33019/cwd (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 33019 root rtd unknown /proc/33019/root (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 33019 root txt unknown /proc/33019/exe (readlink: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:(sd-pam) 33019 root NOFD /proc/33019/fd (opendir: Permission denied) 2024-03-16T12:24:12.274 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34684 root cwd unknown /proc/34684/cwd (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34684 root rtd unknown /proc/34684/root (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34684 root txt unknown /proc/34684/exe (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34684 root NOFD /proc/34684/fd (opendir: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34709 root cwd unknown /proc/34709/cwd (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34709 root rtd unknown /proc/34709/root (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34709 root txt unknown /proc/34709/exe (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:sshd 34709 root NOFD /proc/34709/fd (opendir: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 46633 root cwd unknown /proc/46633/cwd (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 46633 root rtd unknown /proc/46633/root (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 46633 root txt unknown /proc/46633/exe (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 46633 root NOFD /proc/46633/fd (opendir: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 46635 root cwd unknown /proc/46635/cwd (readlink: Permission denied) 2024-03-16T12:24:12.275 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 46635 root rtd unknown /proc/46635/root (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 46635 root txt unknown /proc/46635/exe (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 46635 root NOFD /proc/46635/fd (opendir: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 ceph cwd unknown /proc/46637/cwd (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 ceph rtd unknown /proc/46637/root (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 ceph txt unknown /proc/46637/exe (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 ceph NOFD /proc/46637/fd (opendir: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46638 log ceph cwd unknown /proc/46637/task/46638/cwd (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46638 log ceph rtd unknown /proc/46637/task/46638/root (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46638 log ceph txt unknown /proc/46637/task/46638/exe (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46638 log ceph NOFD /proc/46637/task/46638/fd (opendir: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46640 msgr-work ceph cwd unknown /proc/46637/task/46640/cwd (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46640 msgr-work ceph rtd unknown /proc/46637/task/46640/root (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46640 msgr-work ceph txt unknown /proc/46637/task/46640/exe (readlink: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46640 msgr-work ceph NOFD /proc/46637/task/46640/fd (opendir: Permission denied) 2024-03-16T12:24:12.276 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46641 msgr-work ceph cwd unknown /proc/46637/task/46641/cwd (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46641 msgr-work ceph rtd unknown /proc/46637/task/46641/root (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46641 msgr-work ceph txt unknown /proc/46637/task/46641/exe (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46641 msgr-work ceph NOFD /proc/46637/task/46641/fd (opendir: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46642 msgr-work ceph cwd unknown /proc/46637/task/46642/cwd (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46642 msgr-work ceph rtd unknown /proc/46637/task/46642/root (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46642 msgr-work ceph txt unknown /proc/46637/task/46642/exe (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46642 msgr-work ceph NOFD /proc/46637/task/46642/fd (opendir: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46646 service ceph cwd unknown /proc/46637/task/46646/cwd (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46646 service ceph rtd unknown /proc/46637/task/46646/root (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46646 service ceph txt unknown /proc/46637/task/46646/exe (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46646 service ceph NOFD /proc/46637/task/46646/fd (opendir: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46647 admin_soc ceph cwd unknown /proc/46637/task/46647/cwd (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46647 admin_soc ceph rtd unknown /proc/46637/task/46647/root (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46647 admin_soc ceph txt unknown /proc/46637/task/46647/exe (readlink: Permission denied) 2024-03-16T12:24:12.277 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46647 admin_soc ceph NOFD /proc/46637/task/46647/fd (opendir: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46790 signal_ha ceph cwd unknown /proc/46637/task/46790/cwd (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46790 signal_ha ceph rtd unknown /proc/46637/task/46790/root (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46790 signal_ha ceph txt unknown /proc/46637/task/46790/exe (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46790 signal_ha ceph NOFD /proc/46637/task/46790/fd (opendir: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46791 io_contex ceph cwd unknown /proc/46637/task/46791/cwd (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46791 io_contex ceph rtd unknown /proc/46637/task/46791/root (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46791 io_contex ceph txt unknown /proc/46637/task/46791/exe (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46791 io_contex ceph NOFD /proc/46637/task/46791/fd (opendir: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46792 io_contex ceph cwd unknown /proc/46637/task/46792/cwd (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46792 io_contex ceph rtd unknown /proc/46637/task/46792/root (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46792 io_contex ceph txt unknown /proc/46637/task/46792/exe (readlink: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46792 io_contex ceph NOFD /proc/46637/task/46792/fd (opendir: Permission denied) 2024-03-16T12:24:12.278 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46823 OpHistory ceph cwd unknown /proc/46637/task/46823/cwd (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46823 OpHistory ceph rtd unknown /proc/46637/task/46823/root (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46823 OpHistory ceph txt unknown /proc/46637/task/46823/exe (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46823 OpHistory ceph NOFD /proc/46637/task/46823/fd (opendir: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46825 ceph_time ceph cwd unknown /proc/46637/task/46825/cwd (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46825 ceph_time ceph rtd unknown /proc/46637/task/46825/root (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46825 ceph_time ceph txt unknown /proc/46637/task/46825/exe (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46825 ceph_time ceph NOFD /proc/46637/task/46825/fd (opendir: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46904 ceph-osd ceph cwd unknown /proc/46637/task/46904/cwd (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46904 ceph-osd ceph rtd unknown /proc/46637/task/46904/root (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46904 ceph-osd ceph txt unknown /proc/46637/task/46904/exe (readlink: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46904 ceph-osd ceph NOFD /proc/46637/task/46904/fd (opendir: Permission denied) 2024-03-16T12:24:12.279 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46928 ceph-osd ceph cwd unknown /proc/46637/task/46928/cwd (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46928 ceph-osd ceph rtd unknown /proc/46637/task/46928/root (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46928 ceph-osd ceph txt unknown /proc/46637/task/46928/exe (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46928 ceph-osd ceph NOFD /proc/46637/task/46928/fd (opendir: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46957 ceph-osd ceph cwd unknown /proc/46637/task/46957/cwd (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46957 ceph-osd ceph rtd unknown /proc/46637/task/46957/root (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46957 ceph-osd ceph txt unknown /proc/46637/task/46957/exe (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46957 ceph-osd ceph NOFD /proc/46637/task/46957/fd (opendir: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46966 ceph-osd ceph cwd unknown /proc/46637/task/46966/cwd (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46966 ceph-osd ceph rtd unknown /proc/46637/task/46966/root (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46966 ceph-osd ceph txt unknown /proc/46637/task/46966/exe (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46966 ceph-osd ceph NOFD /proc/46637/task/46966/fd (opendir: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46989 ceph-osd ceph cwd unknown /proc/46637/task/46989/cwd (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46989 ceph-osd ceph rtd unknown /proc/46637/task/46989/root (readlink: Permission denied) 2024-03-16T12:24:12.280 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46989 ceph-osd ceph txt unknown /proc/46637/task/46989/exe (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46989 ceph-osd ceph NOFD /proc/46637/task/46989/fd (opendir: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46993 ceph-osd ceph cwd unknown /proc/46637/task/46993/cwd (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46993 ceph-osd ceph rtd unknown /proc/46637/task/46993/root (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46993 ceph-osd ceph txt unknown /proc/46637/task/46993/exe (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 46993 ceph-osd ceph NOFD /proc/46637/task/46993/fd (opendir: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47012 ceph-osd ceph cwd unknown /proc/46637/task/47012/cwd (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47012 ceph-osd ceph rtd unknown /proc/46637/task/47012/root (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47012 ceph-osd ceph txt unknown /proc/46637/task/47012/exe (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47012 ceph-osd ceph NOFD /proc/46637/task/47012/fd (opendir: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47041 ceph-osd ceph cwd unknown /proc/46637/task/47041/cwd (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47041 ceph-osd ceph rtd unknown /proc/46637/task/47041/root (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47041 ceph-osd ceph txt unknown /proc/46637/task/47041/exe (readlink: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47041 ceph-osd ceph NOFD /proc/46637/task/47041/fd (opendir: Permission denied) 2024-03-16T12:24:12.281 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47042 ceph-osd ceph cwd unknown /proc/46637/task/47042/cwd (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47042 ceph-osd ceph rtd unknown /proc/46637/task/47042/root (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47042 ceph-osd ceph txt unknown /proc/46637/task/47042/exe (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47042 ceph-osd ceph NOFD /proc/46637/task/47042/fd (opendir: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47043 safe_time ceph cwd unknown /proc/46637/task/47043/cwd (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47043 safe_time ceph rtd unknown /proc/46637/task/47043/root (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47043 safe_time ceph txt unknown /proc/46637/task/47043/exe (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47043 safe_time ceph NOFD /proc/46637/task/47043/fd (opendir: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47044 safe_time ceph cwd unknown /proc/46637/task/47044/cwd (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47044 safe_time ceph rtd unknown /proc/46637/task/47044/root (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47044 safe_time ceph txt unknown /proc/46637/task/47044/exe (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47044 safe_time ceph NOFD /proc/46637/task/47044/fd (opendir: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47045 safe_time ceph cwd unknown /proc/46637/task/47045/cwd (readlink: Permission denied) 2024-03-16T12:24:12.282 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47045 safe_time ceph rtd unknown /proc/46637/task/47045/root (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47045 safe_time ceph txt unknown /proc/46637/task/47045/exe (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47045 safe_time ceph NOFD /proc/46637/task/47045/fd (opendir: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47046 safe_time ceph cwd unknown /proc/46637/task/47046/cwd (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47046 safe_time ceph rtd unknown /proc/46637/task/47046/root (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47046 safe_time ceph txt unknown /proc/46637/task/47046/exe (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47046 safe_time ceph NOFD /proc/46637/task/47046/fd (opendir: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47047 fn_anonym ceph cwd unknown /proc/46637/task/47047/cwd (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47047 fn_anonym ceph rtd unknown /proc/46637/task/47047/root (readlink: Permission denied) 2024-03-16T12:24:12.283 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47047 fn_anonym ceph txt unknown /proc/46637/task/47047/exe (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47047 fn_anonym ceph NOFD /proc/46637/task/47047/fd (opendir: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47052 rocksdb:l ceph cwd unknown /proc/46637/task/47052/cwd (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47052 rocksdb:l ceph rtd unknown /proc/46637/task/47052/root (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47052 rocksdb:l ceph txt unknown /proc/46637/task/47052/exe (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47052 rocksdb:l ceph NOFD /proc/46637/task/47052/fd (opendir: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47053 rocksdb:l ceph cwd unknown /proc/46637/task/47053/cwd (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47053 rocksdb:l ceph rtd unknown /proc/46637/task/47053/root (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47053 rocksdb:l ceph txt unknown /proc/46637/task/47053/exe (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47053 rocksdb:l ceph NOFD /proc/46637/task/47053/fd (opendir: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47054 rocksdb:l ceph cwd unknown /proc/46637/task/47054/cwd (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47054 rocksdb:l ceph rtd unknown /proc/46637/task/47054/root (readlink: Permission denied) 2024-03-16T12:24:12.286 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47054 rocksdb:l ceph txt unknown /proc/46637/task/47054/exe (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47054 rocksdb:l ceph NOFD /proc/46637/task/47054/fd (opendir: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47055 rocksdb:h ceph cwd unknown /proc/46637/task/47055/cwd (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47055 rocksdb:h ceph rtd unknown /proc/46637/task/47055/root (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47055 rocksdb:h ceph txt unknown /proc/46637/task/47055/exe (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47055 rocksdb:h ceph NOFD /proc/46637/task/47055/fd (opendir: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47497 bstore_ai ceph cwd unknown /proc/46637/task/47497/cwd (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47497 bstore_ai ceph rtd unknown /proc/46637/task/47497/root (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47497 bstore_ai ceph txt unknown /proc/46637/task/47497/exe (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47497 bstore_ai ceph NOFD /proc/46637/task/47497/fd (opendir: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47705 bstore_ai ceph cwd unknown /proc/46637/task/47705/cwd (readlink: Permission denied) 2024-03-16T12:24:12.287 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47705 bstore_ai ceph rtd unknown /proc/46637/task/47705/root (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47705 bstore_ai ceph txt unknown /proc/46637/task/47705/exe (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47705 bstore_ai ceph NOFD /proc/46637/task/47705/fd (opendir: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47888 ceph-osd ceph cwd unknown /proc/46637/task/47888/cwd (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47888 ceph-osd ceph rtd unknown /proc/46637/task/47888/root (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47888 ceph-osd ceph txt unknown /proc/46637/task/47888/exe (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47888 ceph-osd ceph NOFD /proc/46637/task/47888/fd (opendir: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47889 cfin ceph cwd unknown /proc/46637/task/47889/cwd (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47889 cfin ceph rtd unknown /proc/46637/task/47889/root (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47889 cfin ceph txt unknown /proc/46637/task/47889/exe (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47889 cfin ceph NOFD /proc/46637/task/47889/fd (opendir: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47890 bstore_kv ceph cwd unknown /proc/46637/task/47890/cwd (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47890 bstore_kv ceph rtd unknown /proc/46637/task/47890/root (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47890 bstore_kv ceph txt unknown /proc/46637/task/47890/exe (readlink: Permission denied) 2024-03-16T12:24:12.288 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47890 bstore_kv ceph NOFD /proc/46637/task/47890/fd (opendir: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47891 bstore_kv ceph cwd unknown /proc/46637/task/47891/cwd (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47891 bstore_kv ceph rtd unknown /proc/46637/task/47891/root (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47891 bstore_kv ceph txt unknown /proc/46637/task/47891/exe (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47891 bstore_kv ceph NOFD /proc/46637/task/47891/fd (opendir: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47892 bstore_me ceph cwd unknown /proc/46637/task/47892/cwd (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47892 bstore_me ceph rtd unknown /proc/46637/task/47892/root (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47892 bstore_me ceph txt unknown /proc/46637/task/47892/exe (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47892 bstore_me ceph NOFD /proc/46637/task/47892/fd (opendir: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47893 ms_dispat ceph cwd unknown /proc/46637/task/47893/cwd (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47893 ms_dispat ceph rtd unknown /proc/46637/task/47893/root (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47893 ms_dispat ceph txt unknown /proc/46637/task/47893/exe (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47893 ms_dispat ceph NOFD /proc/46637/task/47893/fd (opendir: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47894 ms_local ceph cwd unknown /proc/46637/task/47894/cwd (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47894 ms_local ceph rtd unknown /proc/46637/task/47894/root (readlink: Permission denied) 2024-03-16T12:24:12.289 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47894 ms_local ceph txt unknown /proc/46637/task/47894/exe (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47894 ms_local ceph NOFD /proc/46637/task/47894/fd (opendir: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47895 safe_time ceph cwd unknown /proc/46637/task/47895/cwd (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47895 safe_time ceph rtd unknown /proc/46637/task/47895/root (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47895 safe_time ceph txt unknown /proc/46637/task/47895/exe (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47895 safe_time ceph NOFD /proc/46637/task/47895/fd (opendir: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47896 safe_time ceph cwd unknown /proc/46637/task/47896/cwd (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47896 safe_time ceph rtd unknown /proc/46637/task/47896/root (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47896 safe_time ceph txt unknown /proc/46637/task/47896/exe (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47896 safe_time ceph NOFD /proc/46637/task/47896/fd (opendir: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47897 ms_dispat ceph cwd unknown /proc/46637/task/47897/cwd (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47897 ms_dispat ceph rtd unknown /proc/46637/task/47897/root (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47897 ms_dispat ceph txt unknown /proc/46637/task/47897/exe (readlink: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47897 ms_dispat ceph NOFD /proc/46637/task/47897/fd (opendir: Permission denied) 2024-03-16T12:24:12.290 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47898 ms_local ceph cwd unknown /proc/46637/task/47898/cwd (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47898 ms_local ceph rtd unknown /proc/46637/task/47898/root (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47898 ms_local ceph txt unknown /proc/46637/task/47898/exe (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47898 ms_local ceph NOFD /proc/46637/task/47898/fd (opendir: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47899 ms_dispat ceph cwd unknown /proc/46637/task/47899/cwd (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47899 ms_dispat ceph rtd unknown /proc/46637/task/47899/root (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47899 ms_dispat ceph txt unknown /proc/46637/task/47899/exe (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47899 ms_dispat ceph NOFD /proc/46637/task/47899/fd (opendir: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47900 ms_local ceph cwd unknown /proc/46637/task/47900/cwd (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47900 ms_local ceph rtd unknown /proc/46637/task/47900/root (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47900 ms_local ceph txt unknown /proc/46637/task/47900/exe (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47900 ms_local ceph NOFD /proc/46637/task/47900/fd (opendir: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47901 ms_dispat ceph cwd unknown /proc/46637/task/47901/cwd (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47901 ms_dispat ceph rtd unknown /proc/46637/task/47901/root (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47901 ms_dispat ceph txt unknown /proc/46637/task/47901/exe (readlink: Permission denied) 2024-03-16T12:24:12.291 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47901 ms_dispat ceph NOFD /proc/46637/task/47901/fd (opendir: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47902 ms_local ceph cwd unknown /proc/46637/task/47902/cwd (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47902 ms_local ceph rtd unknown /proc/46637/task/47902/root (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47902 ms_local ceph txt unknown /proc/46637/task/47902/exe (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47902 ms_local ceph NOFD /proc/46637/task/47902/fd (opendir: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47903 ms_dispat ceph cwd unknown /proc/46637/task/47903/cwd (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47903 ms_dispat ceph rtd unknown /proc/46637/task/47903/root (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47903 ms_dispat ceph txt unknown /proc/46637/task/47903/exe (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47903 ms_dispat ceph NOFD /proc/46637/task/47903/fd (opendir: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47904 ms_local ceph cwd unknown /proc/46637/task/47904/cwd (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47904 ms_local ceph rtd unknown /proc/46637/task/47904/root (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47904 ms_local ceph txt unknown /proc/46637/task/47904/exe (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47904 ms_local ceph NOFD /proc/46637/task/47904/fd (opendir: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47905 ms_dispat ceph cwd unknown /proc/46637/task/47905/cwd (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47905 ms_dispat ceph rtd unknown /proc/46637/task/47905/root (readlink: Permission denied) 2024-03-16T12:24:12.292 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47905 ms_dispat ceph txt unknown /proc/46637/task/47905/exe (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47905 ms_dispat ceph NOFD /proc/46637/task/47905/fd (opendir: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47906 ms_local ceph cwd unknown /proc/46637/task/47906/cwd (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47906 ms_local ceph rtd unknown /proc/46637/task/47906/root (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47906 ms_local ceph txt unknown /proc/46637/task/47906/exe (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47906 ms_local ceph NOFD /proc/46637/task/47906/fd (opendir: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47907 ms_dispat ceph cwd unknown /proc/46637/task/47907/cwd (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47907 ms_dispat ceph rtd unknown /proc/46637/task/47907/root (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47907 ms_dispat ceph txt unknown /proc/46637/task/47907/exe (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47907 ms_dispat ceph NOFD /proc/46637/task/47907/fd (opendir: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47908 ms_local ceph cwd unknown /proc/46637/task/47908/cwd (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47908 ms_local ceph rtd unknown /proc/46637/task/47908/root (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47908 ms_local ceph txt unknown /proc/46637/task/47908/exe (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47908 ms_local ceph NOFD /proc/46637/task/47908/fd (opendir: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47909 fn_anonym ceph cwd unknown /proc/46637/task/47909/cwd (readlink: Permission denied) 2024-03-16T12:24:12.293 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47909 fn_anonym ceph rtd unknown /proc/46637/task/47909/root (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47909 fn_anonym ceph txt unknown /proc/46637/task/47909/exe (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47909 fn_anonym ceph NOFD /proc/46637/task/47909/fd (opendir: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47910 finisher ceph cwd unknown /proc/46637/task/47910/cwd (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47910 finisher ceph rtd unknown /proc/46637/task/47910/root (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47910 finisher ceph txt unknown /proc/46637/task/47910/exe (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47910 finisher ceph NOFD /proc/46637/task/47910/fd (opendir: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47911 safe_time ceph cwd unknown /proc/46637/task/47911/cwd (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47911 safe_time ceph rtd unknown /proc/46637/task/47911/root (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47911 safe_time ceph txt unknown /proc/46637/task/47911/exe (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47911 safe_time ceph NOFD /proc/46637/task/47911/fd (opendir: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47912 safe_time ceph cwd unknown /proc/46637/task/47912/cwd (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47912 safe_time ceph rtd unknown /proc/46637/task/47912/root (readlink: Permission denied) 2024-03-16T12:24:12.294 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47912 safe_time ceph txt unknown /proc/46637/task/47912/exe (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47912 safe_time ceph NOFD /proc/46637/task/47912/fd (opendir: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47913 ceph-osd ceph cwd unknown /proc/46637/task/47913/cwd (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47913 ceph-osd ceph rtd unknown /proc/46637/task/47913/root (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47913 ceph-osd ceph txt unknown /proc/46637/task/47913/exe (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47913 ceph-osd ceph NOFD /proc/46637/task/47913/fd (opendir: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47914 osd_srv_a ceph cwd unknown /proc/46637/task/47914/cwd (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47914 osd_srv_a ceph rtd unknown /proc/46637/task/47914/root (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47914 osd_srv_a ceph txt unknown /proc/46637/task/47914/exe (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47914 osd_srv_a ceph NOFD /proc/46637/task/47914/fd (opendir: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47915 tp_osd_tp ceph cwd unknown /proc/46637/task/47915/cwd (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47915 tp_osd_tp ceph rtd unknown /proc/46637/task/47915/root (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47915 tp_osd_tp ceph txt unknown /proc/46637/task/47915/exe (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47915 tp_osd_tp ceph NOFD /proc/46637/task/47915/fd (opendir: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47916 tp_osd_tp ceph cwd unknown /proc/46637/task/47916/cwd (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47916 tp_osd_tp ceph rtd unknown /proc/46637/task/47916/root (readlink: Permission denied) 2024-03-16T12:24:12.295 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47916 tp_osd_tp ceph txt unknown /proc/46637/task/47916/exe (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47916 tp_osd_tp ceph NOFD /proc/46637/task/47916/fd (opendir: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47917 tp_osd_tp ceph cwd unknown /proc/46637/task/47917/cwd (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47917 tp_osd_tp ceph rtd unknown /proc/46637/task/47917/root (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47917 tp_osd_tp ceph txt unknown /proc/46637/task/47917/exe (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47917 tp_osd_tp ceph NOFD /proc/46637/task/47917/fd (opendir: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47918 tp_osd_tp ceph cwd unknown /proc/46637/task/47918/cwd (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47918 tp_osd_tp ceph rtd unknown /proc/46637/task/47918/root (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47918 tp_osd_tp ceph txt unknown /proc/46637/task/47918/exe (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47918 tp_osd_tp ceph NOFD /proc/46637/task/47918/fd (opendir: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47919 tp_osd_tp ceph cwd unknown /proc/46637/task/47919/cwd (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47919 tp_osd_tp ceph rtd unknown /proc/46637/task/47919/root (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47919 tp_osd_tp ceph txt unknown /proc/46637/task/47919/exe (readlink: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47919 tp_osd_tp ceph NOFD /proc/46637/task/47919/fd (opendir: Permission denied) 2024-03-16T12:24:12.296 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47920 tp_osd_tp ceph cwd unknown /proc/46637/task/47920/cwd (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47920 tp_osd_tp ceph rtd unknown /proc/46637/task/47920/root (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47920 tp_osd_tp ceph txt unknown /proc/46637/task/47920/exe (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47920 tp_osd_tp ceph NOFD /proc/46637/task/47920/fd (opendir: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47921 tp_osd_tp ceph cwd unknown /proc/46637/task/47921/cwd (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47921 tp_osd_tp ceph rtd unknown /proc/46637/task/47921/root (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47921 tp_osd_tp ceph txt unknown /proc/46637/task/47921/exe (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47921 tp_osd_tp ceph NOFD /proc/46637/task/47921/fd (opendir: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47922 tp_osd_tp ceph cwd unknown /proc/46637/task/47922/cwd (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47922 tp_osd_tp ceph rtd unknown /proc/46637/task/47922/root (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47922 tp_osd_tp ceph txt unknown /proc/46637/task/47922/exe (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47922 tp_osd_tp ceph NOFD /proc/46637/task/47922/fd (opendir: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47923 tp_osd_tp ceph cwd unknown /proc/46637/task/47923/cwd (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47923 tp_osd_tp ceph rtd unknown /proc/46637/task/47923/root (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47923 tp_osd_tp ceph txt unknown /proc/46637/task/47923/exe (readlink: Permission denied) 2024-03-16T12:24:12.297 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47923 tp_osd_tp ceph NOFD /proc/46637/task/47923/fd (opendir: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47924 tp_osd_tp ceph cwd unknown /proc/46637/task/47924/cwd (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47924 tp_osd_tp ceph rtd unknown /proc/46637/task/47924/root (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47924 tp_osd_tp ceph txt unknown /proc/46637/task/47924/exe (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47924 tp_osd_tp ceph NOFD /proc/46637/task/47924/fd (opendir: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47925 tp_osd_tp ceph cwd unknown /proc/46637/task/47925/cwd (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47925 tp_osd_tp ceph rtd unknown /proc/46637/task/47925/root (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47925 tp_osd_tp ceph txt unknown /proc/46637/task/47925/exe (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47925 tp_osd_tp ceph NOFD /proc/46637/task/47925/fd (opendir: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47926 tp_osd_tp ceph cwd unknown /proc/46637/task/47926/cwd (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47926 tp_osd_tp ceph rtd unknown /proc/46637/task/47926/root (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47926 tp_osd_tp ceph txt unknown /proc/46637/task/47926/exe (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47926 tp_osd_tp ceph NOFD /proc/46637/task/47926/fd (opendir: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47927 tp_osd_tp ceph cwd unknown /proc/46637/task/47927/cwd (readlink: Permission denied) 2024-03-16T12:24:12.298 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47927 tp_osd_tp ceph rtd unknown /proc/46637/task/47927/root (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47927 tp_osd_tp ceph txt unknown /proc/46637/task/47927/exe (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47927 tp_osd_tp ceph NOFD /proc/46637/task/47927/fd (opendir: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47928 tp_osd_tp ceph cwd unknown /proc/46637/task/47928/cwd (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47928 tp_osd_tp ceph rtd unknown /proc/46637/task/47928/root (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47928 tp_osd_tp ceph txt unknown /proc/46637/task/47928/exe (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47928 tp_osd_tp ceph NOFD /proc/46637/task/47928/fd (opendir: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47929 tp_osd_tp ceph cwd unknown /proc/46637/task/47929/cwd (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47929 tp_osd_tp ceph rtd unknown /proc/46637/task/47929/root (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47929 tp_osd_tp ceph txt unknown /proc/46637/task/47929/exe (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47929 tp_osd_tp ceph NOFD /proc/46637/task/47929/fd (opendir: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47930 tp_osd_tp ceph cwd unknown /proc/46637/task/47930/cwd (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47930 tp_osd_tp ceph rtd unknown /proc/46637/task/47930/root (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47930 tp_osd_tp ceph txt unknown /proc/46637/task/47930/exe (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47930 tp_osd_tp ceph NOFD /proc/46637/task/47930/fd (opendir: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47931 osd_srv_h ceph cwd unknown /proc/46637/task/47931/cwd (readlink: Permission denied) 2024-03-16T12:24:12.299 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47931 osd_srv_h ceph rtd unknown /proc/46637/task/47931/root (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47931 osd_srv_h ceph txt unknown /proc/46637/task/47931/exe (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 46637 47931 osd_srv_h ceph NOFD /proc/46637/task/47931/fd (opendir: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 48036 root cwd unknown /proc/48036/cwd (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 48036 root rtd unknown /proc/48036/root (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 48036 root txt unknown /proc/48036/exe (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 48036 root NOFD /proc/48036/fd (opendir: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 48066 root cwd unknown /proc/48066/cwd (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 48066 root rtd unknown /proc/48066/root (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 48066 root txt unknown /proc/48066/exe (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 48066 root NOFD /proc/48066/fd (opendir: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 50996 root cwd unknown /proc/50996/cwd (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 50996 root rtd unknown /proc/50996/root (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 50996 root txt unknown /proc/50996/exe (readlink: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 50996 root NOFD /proc/50996/fd (opendir: Permission denied) 2024-03-16T12:24:12.300 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 53338 root cwd unknown /proc/53338/cwd (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 53338 root rtd unknown /proc/53338/root (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 53338 root txt unknown /proc/53338/exe (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 53338 root NOFD /proc/53338/fd (opendir: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 53340 root cwd unknown /proc/53340/cwd (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 53340 root rtd unknown /proc/53340/root (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 53340 root txt unknown /proc/53340/exe (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 53340 root NOFD /proc/53340/fd (opendir: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 ceph cwd unknown /proc/53342/cwd (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 ceph rtd unknown /proc/53342/root (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 ceph txt unknown /proc/53342/exe (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 ceph NOFD /proc/53342/fd (opendir: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53343 log ceph cwd unknown /proc/53342/task/53343/cwd (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53343 log ceph rtd unknown /proc/53342/task/53343/root (readlink: Permission denied) 2024-03-16T12:24:12.301 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53343 log ceph txt unknown /proc/53342/task/53343/exe (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53343 log ceph NOFD /proc/53342/task/53343/fd (opendir: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53345 msgr-work ceph cwd unknown /proc/53342/task/53345/cwd (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53345 msgr-work ceph rtd unknown /proc/53342/task/53345/root (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53345 msgr-work ceph txt unknown /proc/53342/task/53345/exe (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53345 msgr-work ceph NOFD /proc/53342/task/53345/fd (opendir: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53346 msgr-work ceph cwd unknown /proc/53342/task/53346/cwd (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53346 msgr-work ceph rtd unknown /proc/53342/task/53346/root (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53346 msgr-work ceph txt unknown /proc/53342/task/53346/exe (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53346 msgr-work ceph NOFD /proc/53342/task/53346/fd (opendir: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53347 msgr-work ceph cwd unknown /proc/53342/task/53347/cwd (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53347 msgr-work ceph rtd unknown /proc/53342/task/53347/root (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53347 msgr-work ceph txt unknown /proc/53342/task/53347/exe (readlink: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53347 msgr-work ceph NOFD /proc/53342/task/53347/fd (opendir: Permission denied) 2024-03-16T12:24:12.302 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53351 service ceph cwd unknown /proc/53342/task/53351/cwd (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53351 service ceph rtd unknown /proc/53342/task/53351/root (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53351 service ceph txt unknown /proc/53342/task/53351/exe (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53351 service ceph NOFD /proc/53342/task/53351/fd (opendir: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53352 admin_soc ceph cwd unknown /proc/53342/task/53352/cwd (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53352 admin_soc ceph rtd unknown /proc/53342/task/53352/root (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53352 admin_soc ceph txt unknown /proc/53342/task/53352/exe (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53352 admin_soc ceph NOFD /proc/53342/task/53352/fd (opendir: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53495 signal_ha ceph cwd unknown /proc/53342/task/53495/cwd (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53495 signal_ha ceph rtd unknown /proc/53342/task/53495/root (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53495 signal_ha ceph txt unknown /proc/53342/task/53495/exe (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53495 signal_ha ceph NOFD /proc/53342/task/53495/fd (opendir: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53496 io_contex ceph cwd unknown /proc/53342/task/53496/cwd (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53496 io_contex ceph rtd unknown /proc/53342/task/53496/root (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53496 io_contex ceph txt unknown /proc/53342/task/53496/exe (readlink: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53496 io_contex ceph NOFD /proc/53342/task/53496/fd (opendir: Permission denied) 2024-03-16T12:24:12.303 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53497 io_contex ceph cwd unknown /proc/53342/task/53497/cwd (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53497 io_contex ceph rtd unknown /proc/53342/task/53497/root (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53497 io_contex ceph txt unknown /proc/53342/task/53497/exe (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53497 io_contex ceph NOFD /proc/53342/task/53497/fd (opendir: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53547 OpHistory ceph cwd unknown /proc/53342/task/53547/cwd (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53547 OpHistory ceph rtd unknown /proc/53342/task/53547/root (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53547 OpHistory ceph txt unknown /proc/53342/task/53547/exe (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53547 OpHistory ceph NOFD /proc/53342/task/53547/fd (opendir: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53548 ceph_time ceph cwd unknown /proc/53342/task/53548/cwd (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53548 ceph_time ceph rtd unknown /proc/53342/task/53548/root (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53548 ceph_time ceph txt unknown /proc/53342/task/53548/exe (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53548 ceph_time ceph NOFD /proc/53342/task/53548/fd (opendir: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53741 ceph-osd ceph cwd unknown /proc/53342/task/53741/cwd (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53741 ceph-osd ceph rtd unknown /proc/53342/task/53741/root (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53741 ceph-osd ceph txt unknown /proc/53342/task/53741/exe (readlink: Permission denied) 2024-03-16T12:24:12.304 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53741 ceph-osd ceph NOFD /proc/53342/task/53741/fd (opendir: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53742 safe_time ceph cwd unknown /proc/53342/task/53742/cwd (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53742 safe_time ceph rtd unknown /proc/53342/task/53742/root (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53742 safe_time ceph txt unknown /proc/53342/task/53742/exe (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53742 safe_time ceph NOFD /proc/53342/task/53742/fd (opendir: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53743 safe_time ceph cwd unknown /proc/53342/task/53743/cwd (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53743 safe_time ceph rtd unknown /proc/53342/task/53743/root (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53743 safe_time ceph txt unknown /proc/53342/task/53743/exe (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53743 safe_time ceph NOFD /proc/53342/task/53743/fd (opendir: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53744 safe_time ceph cwd unknown /proc/53342/task/53744/cwd (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53744 safe_time ceph rtd unknown /proc/53342/task/53744/root (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53744 safe_time ceph txt unknown /proc/53342/task/53744/exe (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53744 safe_time ceph NOFD /proc/53342/task/53744/fd (opendir: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53745 safe_time ceph cwd unknown /proc/53342/task/53745/cwd (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53745 safe_time ceph rtd unknown /proc/53342/task/53745/root (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53745 safe_time ceph txt unknown /proc/53342/task/53745/exe (readlink: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53745 safe_time ceph NOFD /proc/53342/task/53745/fd (opendir: Permission denied) 2024-03-16T12:24:12.305 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53746 fn_anonym ceph cwd unknown /proc/53342/task/53746/cwd (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53746 fn_anonym ceph rtd unknown /proc/53342/task/53746/root (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53746 fn_anonym ceph txt unknown /proc/53342/task/53746/exe (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53746 fn_anonym ceph NOFD /proc/53342/task/53746/fd (opendir: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53752 rocksdb:l ceph cwd unknown /proc/53342/task/53752/cwd (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53752 rocksdb:l ceph rtd unknown /proc/53342/task/53752/root (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53752 rocksdb:l ceph txt unknown /proc/53342/task/53752/exe (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53752 rocksdb:l ceph NOFD /proc/53342/task/53752/fd (opendir: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53753 rocksdb:l ceph cwd unknown /proc/53342/task/53753/cwd (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53753 rocksdb:l ceph rtd unknown /proc/53342/task/53753/root (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53753 rocksdb:l ceph txt unknown /proc/53342/task/53753/exe (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53753 rocksdb:l ceph NOFD /proc/53342/task/53753/fd (opendir: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53754 rocksdb:l ceph cwd unknown /proc/53342/task/53754/cwd (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53754 rocksdb:l ceph rtd unknown /proc/53342/task/53754/root (readlink: Permission denied) 2024-03-16T12:24:12.306 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53754 rocksdb:l ceph txt unknown /proc/53342/task/53754/exe (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53754 rocksdb:l ceph NOFD /proc/53342/task/53754/fd (opendir: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53755 rocksdb:h ceph cwd unknown /proc/53342/task/53755/cwd (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53755 rocksdb:h ceph rtd unknown /proc/53342/task/53755/root (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53755 rocksdb:h ceph txt unknown /proc/53342/task/53755/exe (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 53755 rocksdb:h ceph NOFD /proc/53342/task/53755/fd (opendir: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54200 bstore_ai ceph cwd unknown /proc/53342/task/54200/cwd (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54200 bstore_ai ceph rtd unknown /proc/53342/task/54200/root (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54200 bstore_ai ceph txt unknown /proc/53342/task/54200/exe (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54200 bstore_ai ceph NOFD /proc/53342/task/54200/fd (opendir: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54395 bstore_ai ceph cwd unknown /proc/53342/task/54395/cwd (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54395 bstore_ai ceph rtd unknown /proc/53342/task/54395/root (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54395 bstore_ai ceph txt unknown /proc/53342/task/54395/exe (readlink: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54395 bstore_ai ceph NOFD /proc/53342/task/54395/fd (opendir: Permission denied) 2024-03-16T12:24:12.307 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54581 ceph-osd ceph cwd unknown /proc/53342/task/54581/cwd (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54581 ceph-osd ceph rtd unknown /proc/53342/task/54581/root (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54581 ceph-osd ceph txt unknown /proc/53342/task/54581/exe (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54581 ceph-osd ceph NOFD /proc/53342/task/54581/fd (opendir: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54582 cfin ceph cwd unknown /proc/53342/task/54582/cwd (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54582 cfin ceph rtd unknown /proc/53342/task/54582/root (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54582 cfin ceph txt unknown /proc/53342/task/54582/exe (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54582 cfin ceph NOFD /proc/53342/task/54582/fd (opendir: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54583 bstore_kv ceph cwd unknown /proc/53342/task/54583/cwd (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54583 bstore_kv ceph rtd unknown /proc/53342/task/54583/root (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54583 bstore_kv ceph txt unknown /proc/53342/task/54583/exe (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54583 bstore_kv ceph NOFD /proc/53342/task/54583/fd (opendir: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54584 bstore_kv ceph cwd unknown /proc/53342/task/54584/cwd (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54584 bstore_kv ceph rtd unknown /proc/53342/task/54584/root (readlink: Permission denied) 2024-03-16T12:24:12.308 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54584 bstore_kv ceph txt unknown /proc/53342/task/54584/exe (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54584 bstore_kv ceph NOFD /proc/53342/task/54584/fd (opendir: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54585 bstore_me ceph cwd unknown /proc/53342/task/54585/cwd (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54585 bstore_me ceph rtd unknown /proc/53342/task/54585/root (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54585 bstore_me ceph txt unknown /proc/53342/task/54585/exe (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54585 bstore_me ceph NOFD /proc/53342/task/54585/fd (opendir: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54589 ms_dispat ceph cwd unknown /proc/53342/task/54589/cwd (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54589 ms_dispat ceph rtd unknown /proc/53342/task/54589/root (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54589 ms_dispat ceph txt unknown /proc/53342/task/54589/exe (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54589 ms_dispat ceph NOFD /proc/53342/task/54589/fd (opendir: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54590 ms_local ceph cwd unknown /proc/53342/task/54590/cwd (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54590 ms_local ceph rtd unknown /proc/53342/task/54590/root (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54590 ms_local ceph txt unknown /proc/53342/task/54590/exe (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54590 ms_local ceph NOFD /proc/53342/task/54590/fd (opendir: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54591 safe_time ceph cwd unknown /proc/53342/task/54591/cwd (readlink: Permission denied) 2024-03-16T12:24:12.309 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54591 safe_time ceph rtd unknown /proc/53342/task/54591/root (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54591 safe_time ceph txt unknown /proc/53342/task/54591/exe (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54591 safe_time ceph NOFD /proc/53342/task/54591/fd (opendir: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54592 safe_time ceph cwd unknown /proc/53342/task/54592/cwd (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54592 safe_time ceph rtd unknown /proc/53342/task/54592/root (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54592 safe_time ceph txt unknown /proc/53342/task/54592/exe (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54592 safe_time ceph NOFD /proc/53342/task/54592/fd (opendir: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54593 ms_dispat ceph cwd unknown /proc/53342/task/54593/cwd (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54593 ms_dispat ceph rtd unknown /proc/53342/task/54593/root (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54593 ms_dispat ceph txt unknown /proc/53342/task/54593/exe (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54593 ms_dispat ceph NOFD /proc/53342/task/54593/fd (opendir: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54594 ms_local ceph cwd unknown /proc/53342/task/54594/cwd (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54594 ms_local ceph rtd unknown /proc/53342/task/54594/root (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54594 ms_local ceph txt unknown /proc/53342/task/54594/exe (readlink: Permission denied) 2024-03-16T12:24:12.310 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54594 ms_local ceph NOFD /proc/53342/task/54594/fd (opendir: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54595 ms_dispat ceph cwd unknown /proc/53342/task/54595/cwd (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54595 ms_dispat ceph rtd unknown /proc/53342/task/54595/root (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54595 ms_dispat ceph txt unknown /proc/53342/task/54595/exe (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54595 ms_dispat ceph NOFD /proc/53342/task/54595/fd (opendir: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54596 ms_local ceph cwd unknown /proc/53342/task/54596/cwd (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54596 ms_local ceph rtd unknown /proc/53342/task/54596/root (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54596 ms_local ceph txt unknown /proc/53342/task/54596/exe (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54596 ms_local ceph NOFD /proc/53342/task/54596/fd (opendir: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54597 ms_dispat ceph cwd unknown /proc/53342/task/54597/cwd (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54597 ms_dispat ceph rtd unknown /proc/53342/task/54597/root (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54597 ms_dispat ceph txt unknown /proc/53342/task/54597/exe (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54597 ms_dispat ceph NOFD /proc/53342/task/54597/fd (opendir: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54598 ms_local ceph cwd unknown /proc/53342/task/54598/cwd (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54598 ms_local ceph rtd unknown /proc/53342/task/54598/root (readlink: Permission denied) 2024-03-16T12:24:12.311 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54598 ms_local ceph txt unknown /proc/53342/task/54598/exe (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54598 ms_local ceph NOFD /proc/53342/task/54598/fd (opendir: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54599 ms_dispat ceph cwd unknown /proc/53342/task/54599/cwd (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54599 ms_dispat ceph rtd unknown /proc/53342/task/54599/root (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54599 ms_dispat ceph txt unknown /proc/53342/task/54599/exe (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54599 ms_dispat ceph NOFD /proc/53342/task/54599/fd (opendir: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54600 ms_local ceph cwd unknown /proc/53342/task/54600/cwd (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54600 ms_local ceph rtd unknown /proc/53342/task/54600/root (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54600 ms_local ceph txt unknown /proc/53342/task/54600/exe (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54600 ms_local ceph NOFD /proc/53342/task/54600/fd (opendir: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54601 ms_dispat ceph cwd unknown /proc/53342/task/54601/cwd (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54601 ms_dispat ceph rtd unknown /proc/53342/task/54601/root (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54601 ms_dispat ceph txt unknown /proc/53342/task/54601/exe (readlink: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54601 ms_dispat ceph NOFD /proc/53342/task/54601/fd (opendir: Permission denied) 2024-03-16T12:24:12.312 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54602 ms_local ceph cwd unknown /proc/53342/task/54602/cwd (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54602 ms_local ceph rtd unknown /proc/53342/task/54602/root (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54602 ms_local ceph txt unknown /proc/53342/task/54602/exe (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54602 ms_local ceph NOFD /proc/53342/task/54602/fd (opendir: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54603 ms_dispat ceph cwd unknown /proc/53342/task/54603/cwd (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54603 ms_dispat ceph rtd unknown /proc/53342/task/54603/root (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54603 ms_dispat ceph txt unknown /proc/53342/task/54603/exe (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54603 ms_dispat ceph NOFD /proc/53342/task/54603/fd (opendir: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54604 ms_local ceph cwd unknown /proc/53342/task/54604/cwd (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54604 ms_local ceph rtd unknown /proc/53342/task/54604/root (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54604 ms_local ceph txt unknown /proc/53342/task/54604/exe (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54604 ms_local ceph NOFD /proc/53342/task/54604/fd (opendir: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54605 fn_anonym ceph cwd unknown /proc/53342/task/54605/cwd (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54605 fn_anonym ceph rtd unknown /proc/53342/task/54605/root (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54605 fn_anonym ceph txt unknown /proc/53342/task/54605/exe (readlink: Permission denied) 2024-03-16T12:24:12.313 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54605 fn_anonym ceph NOFD /proc/53342/task/54605/fd (opendir: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54606 finisher ceph cwd unknown /proc/53342/task/54606/cwd (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54606 finisher ceph rtd unknown /proc/53342/task/54606/root (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54606 finisher ceph txt unknown /proc/53342/task/54606/exe (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54606 finisher ceph NOFD /proc/53342/task/54606/fd (opendir: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54607 safe_time ceph cwd unknown /proc/53342/task/54607/cwd (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54607 safe_time ceph rtd unknown /proc/53342/task/54607/root (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54607 safe_time ceph txt unknown /proc/53342/task/54607/exe (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54607 safe_time ceph NOFD /proc/53342/task/54607/fd (opendir: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54608 safe_time ceph cwd unknown /proc/53342/task/54608/cwd (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54608 safe_time ceph rtd unknown /proc/53342/task/54608/root (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54608 safe_time ceph txt unknown /proc/53342/task/54608/exe (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54608 safe_time ceph NOFD /proc/53342/task/54608/fd (opendir: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54609 ceph-osd ceph cwd unknown /proc/53342/task/54609/cwd (readlink: Permission denied) 2024-03-16T12:24:12.314 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54609 ceph-osd ceph rtd unknown /proc/53342/task/54609/root (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54609 ceph-osd ceph txt unknown /proc/53342/task/54609/exe (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54609 ceph-osd ceph NOFD /proc/53342/task/54609/fd (opendir: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54610 osd_srv_a ceph cwd unknown /proc/53342/task/54610/cwd (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54610 osd_srv_a ceph rtd unknown /proc/53342/task/54610/root (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54610 osd_srv_a ceph txt unknown /proc/53342/task/54610/exe (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54610 osd_srv_a ceph NOFD /proc/53342/task/54610/fd (opendir: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54611 tp_osd_tp ceph cwd unknown /proc/53342/task/54611/cwd (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54611 tp_osd_tp ceph rtd unknown /proc/53342/task/54611/root (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54611 tp_osd_tp ceph txt unknown /proc/53342/task/54611/exe (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54611 tp_osd_tp ceph NOFD /proc/53342/task/54611/fd (opendir: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54612 tp_osd_tp ceph cwd unknown /proc/53342/task/54612/cwd (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54612 tp_osd_tp ceph rtd unknown /proc/53342/task/54612/root (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54612 tp_osd_tp ceph txt unknown /proc/53342/task/54612/exe (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54612 tp_osd_tp ceph NOFD /proc/53342/task/54612/fd (opendir: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54613 tp_osd_tp ceph cwd unknown /proc/53342/task/54613/cwd (readlink: Permission denied) 2024-03-16T12:24:12.315 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54613 tp_osd_tp ceph rtd unknown /proc/53342/task/54613/root (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54613 tp_osd_tp ceph txt unknown /proc/53342/task/54613/exe (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54613 tp_osd_tp ceph NOFD /proc/53342/task/54613/fd (opendir: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54614 tp_osd_tp ceph cwd unknown /proc/53342/task/54614/cwd (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54614 tp_osd_tp ceph rtd unknown /proc/53342/task/54614/root (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54614 tp_osd_tp ceph txt unknown /proc/53342/task/54614/exe (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54614 tp_osd_tp ceph NOFD /proc/53342/task/54614/fd (opendir: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54615 tp_osd_tp ceph cwd unknown /proc/53342/task/54615/cwd (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54615 tp_osd_tp ceph rtd unknown /proc/53342/task/54615/root (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54615 tp_osd_tp ceph txt unknown /proc/53342/task/54615/exe (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54615 tp_osd_tp ceph NOFD /proc/53342/task/54615/fd (opendir: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54616 tp_osd_tp ceph cwd unknown /proc/53342/task/54616/cwd (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54616 tp_osd_tp ceph rtd unknown /proc/53342/task/54616/root (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54616 tp_osd_tp ceph txt unknown /proc/53342/task/54616/exe (readlink: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54616 tp_osd_tp ceph NOFD /proc/53342/task/54616/fd (opendir: Permission denied) 2024-03-16T12:24:12.316 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54617 tp_osd_tp ceph cwd unknown /proc/53342/task/54617/cwd (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54617 tp_osd_tp ceph rtd unknown /proc/53342/task/54617/root (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54617 tp_osd_tp ceph txt unknown /proc/53342/task/54617/exe (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54617 tp_osd_tp ceph NOFD /proc/53342/task/54617/fd (opendir: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54618 tp_osd_tp ceph cwd unknown /proc/53342/task/54618/cwd (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54618 tp_osd_tp ceph rtd unknown /proc/53342/task/54618/root (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54618 tp_osd_tp ceph txt unknown /proc/53342/task/54618/exe (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54618 tp_osd_tp ceph NOFD /proc/53342/task/54618/fd (opendir: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54619 tp_osd_tp ceph cwd unknown /proc/53342/task/54619/cwd (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54619 tp_osd_tp ceph rtd unknown /proc/53342/task/54619/root (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54619 tp_osd_tp ceph txt unknown /proc/53342/task/54619/exe (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54619 tp_osd_tp ceph NOFD /proc/53342/task/54619/fd (opendir: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54620 tp_osd_tp ceph cwd unknown /proc/53342/task/54620/cwd (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54620 tp_osd_tp ceph rtd unknown /proc/53342/task/54620/root (readlink: Permission denied) 2024-03-16T12:24:12.317 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54620 tp_osd_tp ceph txt unknown /proc/53342/task/54620/exe (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54620 tp_osd_tp ceph NOFD /proc/53342/task/54620/fd (opendir: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54621 tp_osd_tp ceph cwd unknown /proc/53342/task/54621/cwd (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54621 tp_osd_tp ceph rtd unknown /proc/53342/task/54621/root (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54621 tp_osd_tp ceph txt unknown /proc/53342/task/54621/exe (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54621 tp_osd_tp ceph NOFD /proc/53342/task/54621/fd (opendir: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54622 tp_osd_tp ceph cwd unknown /proc/53342/task/54622/cwd (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54622 tp_osd_tp ceph rtd unknown /proc/53342/task/54622/root (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54622 tp_osd_tp ceph txt unknown /proc/53342/task/54622/exe (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54622 tp_osd_tp ceph NOFD /proc/53342/task/54622/fd (opendir: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54623 tp_osd_tp ceph cwd unknown /proc/53342/task/54623/cwd (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54623 tp_osd_tp ceph rtd unknown /proc/53342/task/54623/root (readlink: Permission denied) 2024-03-16T12:24:12.318 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54623 tp_osd_tp ceph txt unknown /proc/53342/task/54623/exe (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54623 tp_osd_tp ceph NOFD /proc/53342/task/54623/fd (opendir: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54624 tp_osd_tp ceph cwd unknown /proc/53342/task/54624/cwd (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54624 tp_osd_tp ceph rtd unknown /proc/53342/task/54624/root (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54624 tp_osd_tp ceph txt unknown /proc/53342/task/54624/exe (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54624 tp_osd_tp ceph NOFD /proc/53342/task/54624/fd (opendir: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54625 tp_osd_tp ceph cwd unknown /proc/53342/task/54625/cwd (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54625 tp_osd_tp ceph rtd unknown /proc/53342/task/54625/root (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54625 tp_osd_tp ceph txt unknown /proc/53342/task/54625/exe (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54625 tp_osd_tp ceph NOFD /proc/53342/task/54625/fd (opendir: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54626 tp_osd_tp ceph cwd unknown /proc/53342/task/54626/cwd (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54626 tp_osd_tp ceph rtd unknown /proc/53342/task/54626/root (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54626 tp_osd_tp ceph txt unknown /proc/53342/task/54626/exe (readlink: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54626 tp_osd_tp ceph NOFD /proc/53342/task/54626/fd (opendir: Permission denied) 2024-03-16T12:24:12.319 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54627 osd_srv_h ceph cwd unknown /proc/53342/task/54627/cwd (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54627 osd_srv_h ceph rtd unknown /proc/53342/task/54627/root (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54627 osd_srv_h ceph txt unknown /proc/53342/task/54627/exe (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 53342 54627 osd_srv_h ceph NOFD /proc/53342/task/54627/fd (opendir: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 54386 root cwd unknown /proc/54386/cwd (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 54386 root rtd unknown /proc/54386/root (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 54386 root txt unknown /proc/54386/exe (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 54386 root NOFD /proc/54386/fd (opendir: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 54749 root cwd unknown /proc/54749/cwd (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 54749 root rtd unknown /proc/54749/root (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 54749 root txt unknown /proc/54749/exe (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 54749 root NOFD /proc/54749/fd (opendir: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 54767 root cwd unknown /proc/54767/cwd (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 54767 root rtd unknown /proc/54767/root (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 54767 root txt unknown /proc/54767/exe (readlink: Permission denied) 2024-03-16T12:24:12.320 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 54767 root NOFD /proc/54767/fd (opendir: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 59886 root cwd unknown /proc/59886/cwd (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 59886 root rtd unknown /proc/59886/root (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 59886 root txt unknown /proc/59886/exe (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 59886 root NOFD /proc/59886/fd (opendir: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 59888 root cwd unknown /proc/59888/cwd (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 59888 root rtd unknown /proc/59888/root (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 59888 root txt unknown /proc/59888/exe (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 59888 root NOFD /proc/59888/fd (opendir: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 ceph cwd unknown /proc/59890/cwd (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 ceph rtd unknown /proc/59890/root (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 ceph txt unknown /proc/59890/exe (readlink: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 ceph NOFD /proc/59890/fd (opendir: Permission denied) 2024-03-16T12:24:12.321 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59891 log ceph cwd unknown /proc/59890/task/59891/cwd (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59891 log ceph rtd unknown /proc/59890/task/59891/root (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59891 log ceph txt unknown /proc/59890/task/59891/exe (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59891 log ceph NOFD /proc/59890/task/59891/fd (opendir: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59893 msgr-work ceph cwd unknown /proc/59890/task/59893/cwd (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59893 msgr-work ceph rtd unknown /proc/59890/task/59893/root (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59893 msgr-work ceph txt unknown /proc/59890/task/59893/exe (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59893 msgr-work ceph NOFD /proc/59890/task/59893/fd (opendir: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59894 msgr-work ceph cwd unknown /proc/59890/task/59894/cwd (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59894 msgr-work ceph rtd unknown /proc/59890/task/59894/root (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59894 msgr-work ceph txt unknown /proc/59890/task/59894/exe (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59894 msgr-work ceph NOFD /proc/59890/task/59894/fd (opendir: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59895 msgr-work ceph cwd unknown /proc/59890/task/59895/cwd (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59895 msgr-work ceph rtd unknown /proc/59890/task/59895/root (readlink: Permission denied) 2024-03-16T12:24:12.322 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59895 msgr-work ceph txt unknown /proc/59890/task/59895/exe (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59895 msgr-work ceph NOFD /proc/59890/task/59895/fd (opendir: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59899 service ceph cwd unknown /proc/59890/task/59899/cwd (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59899 service ceph rtd unknown /proc/59890/task/59899/root (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59899 service ceph txt unknown /proc/59890/task/59899/exe (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59899 service ceph NOFD /proc/59890/task/59899/fd (opendir: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59900 admin_soc ceph cwd unknown /proc/59890/task/59900/cwd (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59900 admin_soc ceph rtd unknown /proc/59890/task/59900/root (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59900 admin_soc ceph txt unknown /proc/59890/task/59900/exe (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 59900 admin_soc ceph NOFD /proc/59890/task/59900/fd (opendir: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60045 signal_ha ceph cwd unknown /proc/59890/task/60045/cwd (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60045 signal_ha ceph rtd unknown /proc/59890/task/60045/root (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60045 signal_ha ceph txt unknown /proc/59890/task/60045/exe (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60045 signal_ha ceph NOFD /proc/59890/task/60045/fd (opendir: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60046 io_contex ceph cwd unknown /proc/59890/task/60046/cwd (readlink: Permission denied) 2024-03-16T12:24:12.323 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60046 io_contex ceph rtd unknown /proc/59890/task/60046/root (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60046 io_contex ceph txt unknown /proc/59890/task/60046/exe (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60046 io_contex ceph NOFD /proc/59890/task/60046/fd (opendir: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60047 io_contex ceph cwd unknown /proc/59890/task/60047/cwd (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60047 io_contex ceph rtd unknown /proc/59890/task/60047/root (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60047 io_contex ceph txt unknown /proc/59890/task/60047/exe (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60047 io_contex ceph NOFD /proc/59890/task/60047/fd (opendir: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60071 OpHistory ceph cwd unknown /proc/59890/task/60071/cwd (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60071 OpHistory ceph rtd unknown /proc/59890/task/60071/root (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60071 OpHistory ceph txt unknown /proc/59890/task/60071/exe (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60071 OpHistory ceph NOFD /proc/59890/task/60071/fd (opendir: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60072 ceph_time ceph cwd unknown /proc/59890/task/60072/cwd (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60072 ceph_time ceph rtd unknown /proc/59890/task/60072/root (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60072 ceph_time ceph txt unknown /proc/59890/task/60072/exe (readlink: Permission denied) 2024-03-16T12:24:12.324 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60072 ceph_time ceph NOFD /proc/59890/task/60072/fd (opendir: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60164 ceph-osd ceph cwd unknown /proc/59890/task/60164/cwd (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60164 ceph-osd ceph rtd unknown /proc/59890/task/60164/root (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60164 ceph-osd ceph txt unknown /proc/59890/task/60164/exe (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60164 ceph-osd ceph NOFD /proc/59890/task/60164/fd (opendir: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60190 ceph-osd ceph cwd unknown /proc/59890/task/60190/cwd (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60190 ceph-osd ceph rtd unknown /proc/59890/task/60190/root (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60190 ceph-osd ceph txt unknown /proc/59890/task/60190/exe (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60190 ceph-osd ceph NOFD /proc/59890/task/60190/fd (opendir: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60207 ceph-osd ceph cwd unknown /proc/59890/task/60207/cwd (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60207 ceph-osd ceph rtd unknown /proc/59890/task/60207/root (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60207 ceph-osd ceph txt unknown /proc/59890/task/60207/exe (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60207 ceph-osd ceph NOFD /proc/59890/task/60207/fd (opendir: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60219 ceph-osd ceph cwd unknown /proc/59890/task/60219/cwd (readlink: Permission denied) 2024-03-16T12:24:12.325 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60219 ceph-osd ceph rtd unknown /proc/59890/task/60219/root (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60219 ceph-osd ceph txt unknown /proc/59890/task/60219/exe (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60219 ceph-osd ceph NOFD /proc/59890/task/60219/fd (opendir: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60237 ceph-osd ceph cwd unknown /proc/59890/task/60237/cwd (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60237 ceph-osd ceph rtd unknown /proc/59890/task/60237/root (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60237 ceph-osd ceph txt unknown /proc/59890/task/60237/exe (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60237 ceph-osd ceph NOFD /proc/59890/task/60237/fd (opendir: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60242 ceph-osd ceph cwd unknown /proc/59890/task/60242/cwd (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60242 ceph-osd ceph rtd unknown /proc/59890/task/60242/root (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60242 ceph-osd ceph txt unknown /proc/59890/task/60242/exe (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60242 ceph-osd ceph NOFD /proc/59890/task/60242/fd (opendir: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60266 ceph-osd ceph cwd unknown /proc/59890/task/60266/cwd (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60266 ceph-osd ceph rtd unknown /proc/59890/task/60266/root (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60266 ceph-osd ceph txt unknown /proc/59890/task/60266/exe (readlink: Permission denied) 2024-03-16T12:24:12.326 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60266 ceph-osd ceph NOFD /proc/59890/task/60266/fd (opendir: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60290 ceph-osd ceph cwd unknown /proc/59890/task/60290/cwd (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60290 ceph-osd ceph rtd unknown /proc/59890/task/60290/root (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60290 ceph-osd ceph txt unknown /proc/59890/task/60290/exe (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60290 ceph-osd ceph NOFD /proc/59890/task/60290/fd (opendir: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60291 ceph-osd ceph cwd unknown /proc/59890/task/60291/cwd (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60291 ceph-osd ceph rtd unknown /proc/59890/task/60291/root (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60291 ceph-osd ceph txt unknown /proc/59890/task/60291/exe (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60291 ceph-osd ceph NOFD /proc/59890/task/60291/fd (opendir: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60292 safe_time ceph cwd unknown /proc/59890/task/60292/cwd (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60292 safe_time ceph rtd unknown /proc/59890/task/60292/root (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60292 safe_time ceph txt unknown /proc/59890/task/60292/exe (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60292 safe_time ceph NOFD /proc/59890/task/60292/fd (opendir: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60293 safe_time ceph cwd unknown /proc/59890/task/60293/cwd (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60293 safe_time ceph rtd unknown /proc/59890/task/60293/root (readlink: Permission denied) 2024-03-16T12:24:12.327 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60293 safe_time ceph txt unknown /proc/59890/task/60293/exe (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60293 safe_time ceph NOFD /proc/59890/task/60293/fd (opendir: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60294 safe_time ceph cwd unknown /proc/59890/task/60294/cwd (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60294 safe_time ceph rtd unknown /proc/59890/task/60294/root (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60294 safe_time ceph txt unknown /proc/59890/task/60294/exe (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60294 safe_time ceph NOFD /proc/59890/task/60294/fd (opendir: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60295 safe_time ceph cwd unknown /proc/59890/task/60295/cwd (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60295 safe_time ceph rtd unknown /proc/59890/task/60295/root (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60295 safe_time ceph txt unknown /proc/59890/task/60295/exe (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60295 safe_time ceph NOFD /proc/59890/task/60295/fd (opendir: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60296 fn_anonym ceph cwd unknown /proc/59890/task/60296/cwd (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60296 fn_anonym ceph rtd unknown /proc/59890/task/60296/root (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60296 fn_anonym ceph txt unknown /proc/59890/task/60296/exe (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60296 fn_anonym ceph NOFD /proc/59890/task/60296/fd (opendir: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60300 rocksdb:l ceph cwd unknown /proc/59890/task/60300/cwd (readlink: Permission denied) 2024-03-16T12:24:12.328 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60300 rocksdb:l ceph rtd unknown /proc/59890/task/60300/root (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60300 rocksdb:l ceph txt unknown /proc/59890/task/60300/exe (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60300 rocksdb:l ceph NOFD /proc/59890/task/60300/fd (opendir: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60301 rocksdb:l ceph cwd unknown /proc/59890/task/60301/cwd (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60301 rocksdb:l ceph rtd unknown /proc/59890/task/60301/root (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60301 rocksdb:l ceph txt unknown /proc/59890/task/60301/exe (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60301 rocksdb:l ceph NOFD /proc/59890/task/60301/fd (opendir: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60302 rocksdb:l ceph cwd unknown /proc/59890/task/60302/cwd (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60302 rocksdb:l ceph rtd unknown /proc/59890/task/60302/root (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60302 rocksdb:l ceph txt unknown /proc/59890/task/60302/exe (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60302 rocksdb:l ceph NOFD /proc/59890/task/60302/fd (opendir: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60303 rocksdb:h ceph cwd unknown /proc/59890/task/60303/cwd (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60303 rocksdb:h ceph rtd unknown /proc/59890/task/60303/root (readlink: Permission denied) 2024-03-16T12:24:12.329 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60303 rocksdb:h ceph txt unknown /proc/59890/task/60303/exe (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60303 rocksdb:h ceph NOFD /proc/59890/task/60303/fd (opendir: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60687 bstore_ai ceph cwd unknown /proc/59890/task/60687/cwd (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60687 bstore_ai ceph rtd unknown /proc/59890/task/60687/root (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60687 bstore_ai ceph txt unknown /proc/59890/task/60687/exe (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60687 bstore_ai ceph NOFD /proc/59890/task/60687/fd (opendir: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60915 bstore_ai ceph cwd unknown /proc/59890/task/60915/cwd (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60915 bstore_ai ceph rtd unknown /proc/59890/task/60915/root (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60915 bstore_ai ceph txt unknown /proc/59890/task/60915/exe (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 60915 bstore_ai ceph NOFD /proc/59890/task/60915/fd (opendir: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61108 ceph-osd ceph cwd unknown /proc/59890/task/61108/cwd (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61108 ceph-osd ceph rtd unknown /proc/59890/task/61108/root (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61108 ceph-osd ceph txt unknown /proc/59890/task/61108/exe (readlink: Permission denied) 2024-03-16T12:24:12.330 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61108 ceph-osd ceph NOFD /proc/59890/task/61108/fd (opendir: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61109 cfin ceph cwd unknown /proc/59890/task/61109/cwd (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61109 cfin ceph rtd unknown /proc/59890/task/61109/root (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61109 cfin ceph txt unknown /proc/59890/task/61109/exe (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61109 cfin ceph NOFD /proc/59890/task/61109/fd (opendir: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61110 bstore_kv ceph cwd unknown /proc/59890/task/61110/cwd (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61110 bstore_kv ceph rtd unknown /proc/59890/task/61110/root (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61110 bstore_kv ceph txt unknown /proc/59890/task/61110/exe (readlink: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61110 bstore_kv ceph NOFD /proc/59890/task/61110/fd (opendir: Permission denied) 2024-03-16T12:24:12.331 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61111 bstore_kv ceph cwd unknown /proc/59890/task/61111/cwd (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61111 bstore_kv ceph rtd unknown /proc/59890/task/61111/root (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61111 bstore_kv ceph txt unknown /proc/59890/task/61111/exe (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61111 bstore_kv ceph NOFD /proc/59890/task/61111/fd (opendir: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61112 bstore_me ceph cwd unknown /proc/59890/task/61112/cwd (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61112 bstore_me ceph rtd unknown /proc/59890/task/61112/root (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61112 bstore_me ceph txt unknown /proc/59890/task/61112/exe (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61112 bstore_me ceph NOFD /proc/59890/task/61112/fd (opendir: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61119 ms_dispat ceph cwd unknown /proc/59890/task/61119/cwd (readlink: Permission denied) 2024-03-16T12:24:12.332 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61119 ms_dispat ceph rtd unknown /proc/59890/task/61119/root (readlink: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61119 ms_dispat ceph txt unknown /proc/59890/task/61119/exe (readlink: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61119 ms_dispat ceph NOFD /proc/59890/task/61119/fd (opendir: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61120 ms_local ceph cwd unknown /proc/59890/task/61120/cwd (readlink: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61120 ms_local ceph rtd unknown /proc/59890/task/61120/root (readlink: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61120 ms_local ceph txt unknown /proc/59890/task/61120/exe (readlink: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61120 ms_local ceph NOFD /proc/59890/task/61120/fd (opendir: Permission denied) 2024-03-16T12:24:12.333 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61121 safe_time ceph cwd unknown /proc/59890/task/61121/cwd (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61121 safe_time ceph rtd unknown /proc/59890/task/61121/root (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61121 safe_time ceph txt unknown /proc/59890/task/61121/exe (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61121 safe_time ceph NOFD /proc/59890/task/61121/fd (opendir: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61122 safe_time ceph cwd unknown /proc/59890/task/61122/cwd (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61122 safe_time ceph rtd unknown /proc/59890/task/61122/root (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61122 safe_time ceph txt unknown /proc/59890/task/61122/exe (readlink: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61122 safe_time ceph NOFD /proc/59890/task/61122/fd (opendir: Permission denied) 2024-03-16T12:24:12.334 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61123 ms_dispat ceph cwd unknown /proc/59890/task/61123/cwd (readlink: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61123 ms_dispat ceph rtd unknown /proc/59890/task/61123/root (readlink: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61123 ms_dispat ceph txt unknown /proc/59890/task/61123/exe (readlink: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61123 ms_dispat ceph NOFD /proc/59890/task/61123/fd (opendir: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61124 ms_local ceph cwd unknown /proc/59890/task/61124/cwd (readlink: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61124 ms_local ceph rtd unknown /proc/59890/task/61124/root (readlink: Permission denied) 2024-03-16T12:24:12.335 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61124 ms_local ceph txt unknown /proc/59890/task/61124/exe (readlink: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61124 ms_local ceph NOFD /proc/59890/task/61124/fd (opendir: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61125 ms_dispat ceph cwd unknown /proc/59890/task/61125/cwd (readlink: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61125 ms_dispat ceph rtd unknown /proc/59890/task/61125/root (readlink: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61125 ms_dispat ceph txt unknown /proc/59890/task/61125/exe (readlink: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61125 ms_dispat ceph NOFD /proc/59890/task/61125/fd (opendir: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61126 ms_local ceph cwd unknown /proc/59890/task/61126/cwd (readlink: Permission denied) 2024-03-16T12:24:12.336 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61126 ms_local ceph rtd unknown /proc/59890/task/61126/root (readlink: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61126 ms_local ceph txt unknown /proc/59890/task/61126/exe (readlink: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61126 ms_local ceph NOFD /proc/59890/task/61126/fd (opendir: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61127 ms_dispat ceph cwd unknown /proc/59890/task/61127/cwd (readlink: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61127 ms_dispat ceph rtd unknown /proc/59890/task/61127/root (readlink: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61127 ms_dispat ceph txt unknown /proc/59890/task/61127/exe (readlink: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61127 ms_dispat ceph NOFD /proc/59890/task/61127/fd (opendir: Permission denied) 2024-03-16T12:24:12.337 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61128 ms_local ceph cwd unknown /proc/59890/task/61128/cwd (readlink: Permission denied) 2024-03-16T12:24:12.338 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61128 ms_local ceph rtd unknown /proc/59890/task/61128/root (readlink: Permission denied) 2024-03-16T12:24:12.338 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61128 ms_local ceph txt unknown /proc/59890/task/61128/exe (readlink: Permission denied) 2024-03-16T12:24:12.338 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61128 ms_local ceph NOFD /proc/59890/task/61128/fd (opendir: Permission denied) 2024-03-16T12:24:12.338 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61129 ms_dispat ceph cwd unknown /proc/59890/task/61129/cwd (readlink: Permission denied) 2024-03-16T12:24:12.338 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61129 ms_dispat ceph rtd unknown /proc/59890/task/61129/root (readlink: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61129 ms_dispat ceph txt unknown /proc/59890/task/61129/exe (readlink: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61129 ms_dispat ceph NOFD /proc/59890/task/61129/fd (opendir: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61130 ms_local ceph cwd unknown /proc/59890/task/61130/cwd (readlink: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61130 ms_local ceph rtd unknown /proc/59890/task/61130/root (readlink: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61130 ms_local ceph txt unknown /proc/59890/task/61130/exe (readlink: Permission denied) 2024-03-16T12:24:12.339 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61130 ms_local ceph NOFD /proc/59890/task/61130/fd (opendir: Permission denied) 2024-03-16T12:24:12.340 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61131 ms_dispat ceph cwd unknown /proc/59890/task/61131/cwd (readlink: Permission denied) 2024-03-16T12:24:12.340 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61131 ms_dispat ceph rtd unknown /proc/59890/task/61131/root (readlink: Permission denied) 2024-03-16T12:24:12.340 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61131 ms_dispat ceph txt unknown /proc/59890/task/61131/exe (readlink: Permission denied) 2024-03-16T12:24:12.341 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61131 ms_dispat ceph NOFD /proc/59890/task/61131/fd (opendir: Permission denied) 2024-03-16T12:24:12.341 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61132 ms_local ceph cwd unknown /proc/59890/task/61132/cwd (readlink: Permission denied) 2024-03-16T12:24:12.341 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61132 ms_local ceph rtd unknown /proc/59890/task/61132/root (readlink: Permission denied) 2024-03-16T12:24:12.341 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61132 ms_local ceph txt unknown /proc/59890/task/61132/exe (readlink: Permission denied) 2024-03-16T12:24:12.341 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61132 ms_local ceph NOFD /proc/59890/task/61132/fd (opendir: Permission denied) 2024-03-16T12:24:12.342 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61133 ms_dispat ceph cwd unknown /proc/59890/task/61133/cwd (readlink: Permission denied) 2024-03-16T12:24:12.342 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61133 ms_dispat ceph rtd unknown /proc/59890/task/61133/root (readlink: Permission denied) 2024-03-16T12:24:12.342 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61133 ms_dispat ceph txt unknown /proc/59890/task/61133/exe (readlink: Permission denied) 2024-03-16T12:24:12.342 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61133 ms_dispat ceph NOFD /proc/59890/task/61133/fd (opendir: Permission denied) 2024-03-16T12:24:12.342 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61134 ms_local ceph cwd unknown /proc/59890/task/61134/cwd (readlink: Permission denied) 2024-03-16T12:24:12.343 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61134 ms_local ceph rtd unknown /proc/59890/task/61134/root (readlink: Permission denied) 2024-03-16T12:24:12.343 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61134 ms_local ceph txt unknown /proc/59890/task/61134/exe (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61134 ms_local ceph NOFD /proc/59890/task/61134/fd (opendir: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61135 fn_anonym ceph cwd unknown /proc/59890/task/61135/cwd (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61135 fn_anonym ceph rtd unknown /proc/59890/task/61135/root (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61135 fn_anonym ceph txt unknown /proc/59890/task/61135/exe (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61135 fn_anonym ceph NOFD /proc/59890/task/61135/fd (opendir: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61136 finisher ceph cwd unknown /proc/59890/task/61136/cwd (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61136 finisher ceph rtd unknown /proc/59890/task/61136/root (readlink: Permission denied) 2024-03-16T12:24:12.344 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61136 finisher ceph txt unknown /proc/59890/task/61136/exe (readlink: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61136 finisher ceph NOFD /proc/59890/task/61136/fd (opendir: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61137 safe_time ceph cwd unknown /proc/59890/task/61137/cwd (readlink: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61137 safe_time ceph rtd unknown /proc/59890/task/61137/root (readlink: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61137 safe_time ceph txt unknown /proc/59890/task/61137/exe (readlink: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61137 safe_time ceph NOFD /proc/59890/task/61137/fd (opendir: Permission denied) 2024-03-16T12:24:12.345 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61138 safe_time ceph cwd unknown /proc/59890/task/61138/cwd (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61138 safe_time ceph rtd unknown /proc/59890/task/61138/root (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61138 safe_time ceph txt unknown /proc/59890/task/61138/exe (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61138 safe_time ceph NOFD /proc/59890/task/61138/fd (opendir: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61139 ceph-osd ceph cwd unknown /proc/59890/task/61139/cwd (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61139 ceph-osd ceph rtd unknown /proc/59890/task/61139/root (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61139 ceph-osd ceph txt unknown /proc/59890/task/61139/exe (readlink: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61139 ceph-osd ceph NOFD /proc/59890/task/61139/fd (opendir: Permission denied) 2024-03-16T12:24:12.346 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61140 osd_srv_a ceph cwd unknown /proc/59890/task/61140/cwd (readlink: Permission denied) 2024-03-16T12:24:12.347 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61140 osd_srv_a ceph rtd unknown /proc/59890/task/61140/root (readlink: Permission denied) 2024-03-16T12:24:12.348 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61140 osd_srv_a ceph txt unknown /proc/59890/task/61140/exe (readlink: Permission denied) 2024-03-16T12:24:12.348 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61140 osd_srv_a ceph NOFD /proc/59890/task/61140/fd (opendir: Permission denied) 2024-03-16T12:24:12.348 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61141 tp_osd_tp ceph cwd unknown /proc/59890/task/61141/cwd (readlink: Permission denied) 2024-03-16T12:24:12.348 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61141 tp_osd_tp ceph rtd unknown /proc/59890/task/61141/root (readlink: Permission denied) 2024-03-16T12:24:12.348 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61141 tp_osd_tp ceph txt unknown /proc/59890/task/61141/exe (readlink: Permission denied) 2024-03-16T12:24:12.349 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61141 tp_osd_tp ceph NOFD /proc/59890/task/61141/fd (opendir: Permission denied) 2024-03-16T12:24:12.349 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61142 tp_osd_tp ceph cwd unknown /proc/59890/task/61142/cwd (readlink: Permission denied) 2024-03-16T12:24:12.349 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61142 tp_osd_tp ceph rtd unknown /proc/59890/task/61142/root (readlink: Permission denied) 2024-03-16T12:24:12.350 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61142 tp_osd_tp ceph txt unknown /proc/59890/task/61142/exe (readlink: Permission denied) 2024-03-16T12:24:12.350 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61142 tp_osd_tp ceph NOFD /proc/59890/task/61142/fd (opendir: Permission denied) 2024-03-16T12:24:12.350 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61143 tp_osd_tp ceph cwd unknown /proc/59890/task/61143/cwd (readlink: Permission denied) 2024-03-16T12:24:12.350 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61143 tp_osd_tp ceph rtd unknown /proc/59890/task/61143/root (readlink: Permission denied) 2024-03-16T12:24:12.350 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61143 tp_osd_tp ceph txt unknown /proc/59890/task/61143/exe (readlink: Permission denied) 2024-03-16T12:24:12.351 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61143 tp_osd_tp ceph NOFD /proc/59890/task/61143/fd (opendir: Permission denied) 2024-03-16T12:24:12.351 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61144 tp_osd_tp ceph cwd unknown /proc/59890/task/61144/cwd (readlink: Permission denied) 2024-03-16T12:24:12.351 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61144 tp_osd_tp ceph rtd unknown /proc/59890/task/61144/root (readlink: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61144 tp_osd_tp ceph txt unknown /proc/59890/task/61144/exe (readlink: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61144 tp_osd_tp ceph NOFD /proc/59890/task/61144/fd (opendir: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61145 tp_osd_tp ceph cwd unknown /proc/59890/task/61145/cwd (readlink: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61145 tp_osd_tp ceph rtd unknown /proc/59890/task/61145/root (readlink: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61145 tp_osd_tp ceph txt unknown /proc/59890/task/61145/exe (readlink: Permission denied) 2024-03-16T12:24:12.352 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61145 tp_osd_tp ceph NOFD /proc/59890/task/61145/fd (opendir: Permission denied) 2024-03-16T12:24:12.353 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61146 tp_osd_tp ceph cwd unknown /proc/59890/task/61146/cwd (readlink: Permission denied) 2024-03-16T12:24:12.353 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61146 tp_osd_tp ceph rtd unknown /proc/59890/task/61146/root (readlink: Permission denied) 2024-03-16T12:24:12.353 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61146 tp_osd_tp ceph txt unknown /proc/59890/task/61146/exe (readlink: Permission denied) 2024-03-16T12:24:12.354 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61146 tp_osd_tp ceph NOFD /proc/59890/task/61146/fd (opendir: Permission denied) 2024-03-16T12:24:12.354 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61147 tp_osd_tp ceph cwd unknown /proc/59890/task/61147/cwd (readlink: Permission denied) 2024-03-16T12:24:12.354 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61147 tp_osd_tp ceph rtd unknown /proc/59890/task/61147/root (readlink: Permission denied) 2024-03-16T12:24:12.354 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61147 tp_osd_tp ceph txt unknown /proc/59890/task/61147/exe (readlink: Permission denied) 2024-03-16T12:24:12.354 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61147 tp_osd_tp ceph NOFD /proc/59890/task/61147/fd (opendir: Permission denied) 2024-03-16T12:24:12.355 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61148 tp_osd_tp ceph cwd unknown /proc/59890/task/61148/cwd (readlink: Permission denied) 2024-03-16T12:24:12.355 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61148 tp_osd_tp ceph rtd unknown /proc/59890/task/61148/root (readlink: Permission denied) 2024-03-16T12:24:12.356 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61148 tp_osd_tp ceph txt unknown /proc/59890/task/61148/exe (readlink: Permission denied) 2024-03-16T12:24:12.356 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61148 tp_osd_tp ceph NOFD /proc/59890/task/61148/fd (opendir: Permission denied) 2024-03-16T12:24:12.356 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61149 tp_osd_tp ceph cwd unknown /proc/59890/task/61149/cwd (readlink: Permission denied) 2024-03-16T12:24:12.356 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61149 tp_osd_tp ceph rtd unknown /proc/59890/task/61149/root (readlink: Permission denied) 2024-03-16T12:24:12.357 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61149 tp_osd_tp ceph txt unknown /proc/59890/task/61149/exe (readlink: Permission denied) 2024-03-16T12:24:12.357 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61149 tp_osd_tp ceph NOFD /proc/59890/task/61149/fd (opendir: Permission denied) 2024-03-16T12:24:12.357 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61150 tp_osd_tp ceph cwd unknown /proc/59890/task/61150/cwd (readlink: Permission denied) 2024-03-16T12:24:12.357 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61150 tp_osd_tp ceph rtd unknown /proc/59890/task/61150/root (readlink: Permission denied) 2024-03-16T12:24:12.358 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61150 tp_osd_tp ceph txt unknown /proc/59890/task/61150/exe (readlink: Permission denied) 2024-03-16T12:24:12.358 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61150 tp_osd_tp ceph NOFD /proc/59890/task/61150/fd (opendir: Permission denied) 2024-03-16T12:24:12.359 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61151 tp_osd_tp ceph cwd unknown /proc/59890/task/61151/cwd (readlink: Permission denied) 2024-03-16T12:24:12.359 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61151 tp_osd_tp ceph rtd unknown /proc/59890/task/61151/root (readlink: Permission denied) 2024-03-16T12:24:12.359 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61151 tp_osd_tp ceph txt unknown /proc/59890/task/61151/exe (readlink: Permission denied) 2024-03-16T12:24:12.359 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61151 tp_osd_tp ceph NOFD /proc/59890/task/61151/fd (opendir: Permission denied) 2024-03-16T12:24:12.360 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61152 tp_osd_tp ceph cwd unknown /proc/59890/task/61152/cwd (readlink: Permission denied) 2024-03-16T12:24:12.360 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61152 tp_osd_tp ceph rtd unknown /proc/59890/task/61152/root (readlink: Permission denied) 2024-03-16T12:24:12.361 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61152 tp_osd_tp ceph txt unknown /proc/59890/task/61152/exe (readlink: Permission denied) 2024-03-16T12:24:12.362 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61152 tp_osd_tp ceph NOFD /proc/59890/task/61152/fd (opendir: Permission denied) 2024-03-16T12:24:12.362 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61153 tp_osd_tp ceph cwd unknown /proc/59890/task/61153/cwd (readlink: Permission denied) 2024-03-16T12:24:12.362 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61153 tp_osd_tp ceph rtd unknown /proc/59890/task/61153/root (readlink: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61153 tp_osd_tp ceph txt unknown /proc/59890/task/61153/exe (readlink: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61153 tp_osd_tp ceph NOFD /proc/59890/task/61153/fd (opendir: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61154 tp_osd_tp ceph cwd unknown /proc/59890/task/61154/cwd (readlink: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61154 tp_osd_tp ceph rtd unknown /proc/59890/task/61154/root (readlink: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61154 tp_osd_tp ceph txt unknown /proc/59890/task/61154/exe (readlink: Permission denied) 2024-03-16T12:24:12.363 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61154 tp_osd_tp ceph NOFD /proc/59890/task/61154/fd (opendir: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61155 tp_osd_tp ceph cwd unknown /proc/59890/task/61155/cwd (readlink: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61155 tp_osd_tp ceph rtd unknown /proc/59890/task/61155/root (readlink: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61155 tp_osd_tp ceph txt unknown /proc/59890/task/61155/exe (readlink: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61155 tp_osd_tp ceph NOFD /proc/59890/task/61155/fd (opendir: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61156 tp_osd_tp ceph cwd unknown /proc/59890/task/61156/cwd (readlink: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61156 tp_osd_tp ceph rtd unknown /proc/59890/task/61156/root (readlink: Permission denied) 2024-03-16T12:24:12.364 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61156 tp_osd_tp ceph txt unknown /proc/59890/task/61156/exe (readlink: Permission denied) 2024-03-16T12:24:12.365 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61156 tp_osd_tp ceph NOFD /proc/59890/task/61156/fd (opendir: Permission denied) 2024-03-16T12:24:12.365 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61157 osd_srv_h ceph cwd unknown /proc/59890/task/61157/cwd (readlink: Permission denied) 2024-03-16T12:24:12.365 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61157 osd_srv_h ceph rtd unknown /proc/59890/task/61157/root (readlink: Permission denied) 2024-03-16T12:24:12.365 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61157 osd_srv_h ceph txt unknown /proc/59890/task/61157/exe (readlink: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 59890 61157 osd_srv_h ceph NOFD /proc/59890/task/61157/fd (opendir: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 61281 root cwd unknown /proc/61281/cwd (readlink: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 61281 root rtd unknown /proc/61281/root (readlink: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 61281 root txt unknown /proc/61281/exe (readlink: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 61281 root NOFD /proc/61281/fd (opendir: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 61322 root cwd unknown /proc/61322/cwd (readlink: Permission denied) 2024-03-16T12:24:12.366 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 61322 root rtd unknown /proc/61322/root (readlink: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 61322 root txt unknown /proc/61322/exe (readlink: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 61322 root NOFD /proc/61322/fd (opendir: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 61339 root cwd unknown /proc/61339/cwd (readlink: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 61339 root rtd unknown /proc/61339/root (readlink: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 61339 root txt unknown /proc/61339/exe (readlink: Permission denied) 2024-03-16T12:24:12.367 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 61339 root NOFD /proc/61339/fd (opendir: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 64158 root cwd unknown /proc/64158/cwd (readlink: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 64158 root rtd unknown /proc/64158/root (readlink: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 64158 root txt unknown /proc/64158/exe (readlink: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 64158 root NOFD /proc/64158/fd (opendir: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 65550 root cwd unknown /proc/65550/cwd (readlink: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 65550 root rtd unknown /proc/65550/root (readlink: Permission denied) 2024-03-16T12:24:12.368 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 65550 root txt unknown /proc/65550/exe (readlink: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 65550 root NOFD /proc/65550/fd (opendir: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 66956 root cwd unknown /proc/66956/cwd (readlink: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 66956 root rtd unknown /proc/66956/root (readlink: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 66956 root txt unknown /proc/66956/exe (readlink: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 66956 root NOFD /proc/66956/fd (opendir: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 66958 root cwd unknown /proc/66958/cwd (readlink: Permission denied) 2024-03-16T12:24:12.369 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 66958 root rtd unknown /proc/66958/root (readlink: Permission denied) 2024-03-16T12:24:12.370 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 66958 root txt unknown /proc/66958/exe (readlink: Permission denied) 2024-03-16T12:24:12.370 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 66958 root NOFD /proc/66958/fd (opendir: Permission denied) 2024-03-16T12:24:12.370 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 ceph cwd unknown /proc/66960/cwd (readlink: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 ceph rtd unknown /proc/66960/root (readlink: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 ceph txt unknown /proc/66960/exe (readlink: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 ceph NOFD /proc/66960/fd (opendir: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66961 log ceph cwd unknown /proc/66960/task/66961/cwd (readlink: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66961 log ceph rtd unknown /proc/66960/task/66961/root (readlink: Permission denied) 2024-03-16T12:24:12.371 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66961 log ceph txt unknown /proc/66960/task/66961/exe (readlink: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66961 log ceph NOFD /proc/66960/task/66961/fd (opendir: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66963 msgr-work ceph cwd unknown /proc/66960/task/66963/cwd (readlink: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66963 msgr-work ceph rtd unknown /proc/66960/task/66963/root (readlink: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66963 msgr-work ceph txt unknown /proc/66960/task/66963/exe (readlink: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66963 msgr-work ceph NOFD /proc/66960/task/66963/fd (opendir: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66964 msgr-work ceph cwd unknown /proc/66960/task/66964/cwd (readlink: Permission denied) 2024-03-16T12:24:12.372 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66964 msgr-work ceph rtd unknown /proc/66960/task/66964/root (readlink: Permission denied) 2024-03-16T12:24:12.373 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66964 msgr-work ceph txt unknown /proc/66960/task/66964/exe (readlink: Permission denied) 2024-03-16T12:24:12.373 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66964 msgr-work ceph NOFD /proc/66960/task/66964/fd (opendir: Permission denied) 2024-03-16T12:24:12.373 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66965 msgr-work ceph cwd unknown /proc/66960/task/66965/cwd (readlink: Permission denied) 2024-03-16T12:24:12.373 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66965 msgr-work ceph rtd unknown /proc/66960/task/66965/root (readlink: Permission denied) 2024-03-16T12:24:12.373 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66965 msgr-work ceph txt unknown /proc/66960/task/66965/exe (readlink: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66965 msgr-work ceph NOFD /proc/66960/task/66965/fd (opendir: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66969 service ceph cwd unknown /proc/66960/task/66969/cwd (readlink: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66969 service ceph rtd unknown /proc/66960/task/66969/root (readlink: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66969 service ceph txt unknown /proc/66960/task/66969/exe (readlink: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66969 service ceph NOFD /proc/66960/task/66969/fd (opendir: Permission denied) 2024-03-16T12:24:12.374 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66970 admin_soc ceph cwd unknown /proc/66960/task/66970/cwd (readlink: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66970 admin_soc ceph rtd unknown /proc/66960/task/66970/root (readlink: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66970 admin_soc ceph txt unknown /proc/66960/task/66970/exe (readlink: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 66970 admin_soc ceph NOFD /proc/66960/task/66970/fd (opendir: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67114 signal_ha ceph cwd unknown /proc/66960/task/67114/cwd (readlink: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67114 signal_ha ceph rtd unknown /proc/66960/task/67114/root (readlink: Permission denied) 2024-03-16T12:24:12.375 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67114 signal_ha ceph txt unknown /proc/66960/task/67114/exe (readlink: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67114 signal_ha ceph NOFD /proc/66960/task/67114/fd (opendir: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67115 io_contex ceph cwd unknown /proc/66960/task/67115/cwd (readlink: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67115 io_contex ceph rtd unknown /proc/66960/task/67115/root (readlink: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67115 io_contex ceph txt unknown /proc/66960/task/67115/exe (readlink: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67115 io_contex ceph NOFD /proc/66960/task/67115/fd (opendir: Permission denied) 2024-03-16T12:24:12.376 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67116 io_contex ceph cwd unknown /proc/66960/task/67116/cwd (readlink: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67116 io_contex ceph rtd unknown /proc/66960/task/67116/root (readlink: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67116 io_contex ceph txt unknown /proc/66960/task/67116/exe (readlink: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67116 io_contex ceph NOFD /proc/66960/task/67116/fd (opendir: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67122 OpHistory ceph cwd unknown /proc/66960/task/67122/cwd (readlink: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67122 OpHistory ceph rtd unknown /proc/66960/task/67122/root (readlink: Permission denied) 2024-03-16T12:24:12.377 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67122 OpHistory ceph txt unknown /proc/66960/task/67122/exe (readlink: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67122 OpHistory ceph NOFD /proc/66960/task/67122/fd (opendir: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67123 ceph_time ceph cwd unknown /proc/66960/task/67123/cwd (readlink: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67123 ceph_time ceph rtd unknown /proc/66960/task/67123/root (readlink: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67123 ceph_time ceph txt unknown /proc/66960/task/67123/exe (readlink: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67123 ceph_time ceph NOFD /proc/66960/task/67123/fd (opendir: Permission denied) 2024-03-16T12:24:12.378 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67358 ceph-osd ceph cwd unknown /proc/66960/task/67358/cwd (readlink: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67358 ceph-osd ceph rtd unknown /proc/66960/task/67358/root (readlink: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67358 ceph-osd ceph txt unknown /proc/66960/task/67358/exe (readlink: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67358 ceph-osd ceph NOFD /proc/66960/task/67358/fd (opendir: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67359 safe_time ceph cwd unknown /proc/66960/task/67359/cwd (readlink: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67359 safe_time ceph rtd unknown /proc/66960/task/67359/root (readlink: Permission denied) 2024-03-16T12:24:12.379 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67359 safe_time ceph txt unknown /proc/66960/task/67359/exe (readlink: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67359 safe_time ceph NOFD /proc/66960/task/67359/fd (opendir: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67360 safe_time ceph cwd unknown /proc/66960/task/67360/cwd (readlink: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67360 safe_time ceph rtd unknown /proc/66960/task/67360/root (readlink: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67360 safe_time ceph txt unknown /proc/66960/task/67360/exe (readlink: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67360 safe_time ceph NOFD /proc/66960/task/67360/fd (opendir: Permission denied) 2024-03-16T12:24:12.380 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67361 safe_time ceph cwd unknown /proc/66960/task/67361/cwd (readlink: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67361 safe_time ceph rtd unknown /proc/66960/task/67361/root (readlink: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67361 safe_time ceph txt unknown /proc/66960/task/67361/exe (readlink: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67361 safe_time ceph NOFD /proc/66960/task/67361/fd (opendir: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67362 safe_time ceph cwd unknown /proc/66960/task/67362/cwd (readlink: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67362 safe_time ceph rtd unknown /proc/66960/task/67362/root (readlink: Permission denied) 2024-03-16T12:24:12.381 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67362 safe_time ceph txt unknown /proc/66960/task/67362/exe (readlink: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67362 safe_time ceph NOFD /proc/66960/task/67362/fd (opendir: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67363 fn_anonym ceph cwd unknown /proc/66960/task/67363/cwd (readlink: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67363 fn_anonym ceph rtd unknown /proc/66960/task/67363/root (readlink: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67363 fn_anonym ceph txt unknown /proc/66960/task/67363/exe (readlink: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67363 fn_anonym ceph NOFD /proc/66960/task/67363/fd (opendir: Permission denied) 2024-03-16T12:24:12.382 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67368 rocksdb:l ceph cwd unknown /proc/66960/task/67368/cwd (readlink: Permission denied) 2024-03-16T12:24:12.383 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67368 rocksdb:l ceph rtd unknown /proc/66960/task/67368/root (readlink: Permission denied) 2024-03-16T12:24:12.383 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67368 rocksdb:l ceph txt unknown /proc/66960/task/67368/exe (readlink: Permission denied) 2024-03-16T12:24:12.383 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67368 rocksdb:l ceph NOFD /proc/66960/task/67368/fd (opendir: Permission denied) 2024-03-16T12:24:12.383 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67369 rocksdb:l ceph cwd unknown /proc/66960/task/67369/cwd (readlink: Permission denied) 2024-03-16T12:24:12.383 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67369 rocksdb:l ceph rtd unknown /proc/66960/task/67369/root (readlink: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67369 rocksdb:l ceph txt unknown /proc/66960/task/67369/exe (readlink: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67369 rocksdb:l ceph NOFD /proc/66960/task/67369/fd (opendir: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67370 rocksdb:l ceph cwd unknown /proc/66960/task/67370/cwd (readlink: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67370 rocksdb:l ceph rtd unknown /proc/66960/task/67370/root (readlink: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67370 rocksdb:l ceph txt unknown /proc/66960/task/67370/exe (readlink: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67370 rocksdb:l ceph NOFD /proc/66960/task/67370/fd (opendir: Permission denied) 2024-03-16T12:24:12.384 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67371 rocksdb:h ceph cwd unknown /proc/66960/task/67371/cwd (readlink: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67371 rocksdb:h ceph rtd unknown /proc/66960/task/67371/root (readlink: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67371 rocksdb:h ceph txt unknown /proc/66960/task/67371/exe (readlink: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67371 rocksdb:h ceph NOFD /proc/66960/task/67371/fd (opendir: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67819 bstore_ai ceph cwd unknown /proc/66960/task/67819/cwd (readlink: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67819 bstore_ai ceph rtd unknown /proc/66960/task/67819/root (readlink: Permission denied) 2024-03-16T12:24:12.385 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67819 bstore_ai ceph txt unknown /proc/66960/task/67819/exe (readlink: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 67819 bstore_ai ceph NOFD /proc/66960/task/67819/fd (opendir: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68083 bstore_ai ceph cwd unknown /proc/66960/task/68083/cwd (readlink: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68083 bstore_ai ceph rtd unknown /proc/66960/task/68083/root (readlink: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68083 bstore_ai ceph txt unknown /proc/66960/task/68083/exe (readlink: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68083 bstore_ai ceph NOFD /proc/66960/task/68083/fd (opendir: Permission denied) 2024-03-16T12:24:12.386 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68268 ceph-osd ceph cwd unknown /proc/66960/task/68268/cwd (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68268 ceph-osd ceph rtd unknown /proc/66960/task/68268/root (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68268 ceph-osd ceph txt unknown /proc/66960/task/68268/exe (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68268 ceph-osd ceph NOFD /proc/66960/task/68268/fd (opendir: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68269 cfin ceph cwd unknown /proc/66960/task/68269/cwd (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68269 cfin ceph rtd unknown /proc/66960/task/68269/root (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68269 cfin ceph txt unknown /proc/66960/task/68269/exe (readlink: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68269 cfin ceph NOFD /proc/66960/task/68269/fd (opendir: Permission denied) 2024-03-16T12:24:12.387 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68270 bstore_kv ceph cwd unknown /proc/66960/task/68270/cwd (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68270 bstore_kv ceph rtd unknown /proc/66960/task/68270/root (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68270 bstore_kv ceph txt unknown /proc/66960/task/68270/exe (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68270 bstore_kv ceph NOFD /proc/66960/task/68270/fd (opendir: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68271 bstore_kv ceph cwd unknown /proc/66960/task/68271/cwd (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68271 bstore_kv ceph rtd unknown /proc/66960/task/68271/root (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68271 bstore_kv ceph txt unknown /proc/66960/task/68271/exe (readlink: Permission denied) 2024-03-16T12:24:12.388 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68271 bstore_kv ceph NOFD /proc/66960/task/68271/fd (opendir: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68272 bstore_me ceph cwd unknown /proc/66960/task/68272/cwd (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68272 bstore_me ceph rtd unknown /proc/66960/task/68272/root (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68272 bstore_me ceph txt unknown /proc/66960/task/68272/exe (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68272 bstore_me ceph NOFD /proc/66960/task/68272/fd (opendir: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68274 ms_dispat ceph cwd unknown /proc/66960/task/68274/cwd (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68274 ms_dispat ceph rtd unknown /proc/66960/task/68274/root (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68274 ms_dispat ceph txt unknown /proc/66960/task/68274/exe (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68274 ms_dispat ceph NOFD /proc/66960/task/68274/fd (opendir: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68275 ms_local ceph cwd unknown /proc/66960/task/68275/cwd (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68275 ms_local ceph rtd unknown /proc/66960/task/68275/root (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68275 ms_local ceph txt unknown /proc/66960/task/68275/exe (readlink: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68275 ms_local ceph NOFD /proc/66960/task/68275/fd (opendir: Permission denied) 2024-03-16T12:24:12.389 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68276 safe_time ceph cwd unknown /proc/66960/task/68276/cwd (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68276 safe_time ceph rtd unknown /proc/66960/task/68276/root (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68276 safe_time ceph txt unknown /proc/66960/task/68276/exe (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68276 safe_time ceph NOFD /proc/66960/task/68276/fd (opendir: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68277 safe_time ceph cwd unknown /proc/66960/task/68277/cwd (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68277 safe_time ceph rtd unknown /proc/66960/task/68277/root (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68277 safe_time ceph txt unknown /proc/66960/task/68277/exe (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68277 safe_time ceph NOFD /proc/66960/task/68277/fd (opendir: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68278 ms_dispat ceph cwd unknown /proc/66960/task/68278/cwd (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68278 ms_dispat ceph rtd unknown /proc/66960/task/68278/root (readlink: Permission denied) 2024-03-16T12:24:12.390 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68278 ms_dispat ceph txt unknown /proc/66960/task/68278/exe (readlink: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68278 ms_dispat ceph NOFD /proc/66960/task/68278/fd (opendir: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68279 ms_local ceph cwd unknown /proc/66960/task/68279/cwd (readlink: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68279 ms_local ceph rtd unknown /proc/66960/task/68279/root (readlink: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68279 ms_local ceph txt unknown /proc/66960/task/68279/exe (readlink: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68279 ms_local ceph NOFD /proc/66960/task/68279/fd (opendir: Permission denied) 2024-03-16T12:24:12.391 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68280 ms_dispat ceph cwd unknown /proc/66960/task/68280/cwd (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68280 ms_dispat ceph rtd unknown /proc/66960/task/68280/root (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68280 ms_dispat ceph txt unknown /proc/66960/task/68280/exe (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68280 ms_dispat ceph NOFD /proc/66960/task/68280/fd (opendir: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68281 ms_local ceph cwd unknown /proc/66960/task/68281/cwd (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68281 ms_local ceph rtd unknown /proc/66960/task/68281/root (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68281 ms_local ceph txt unknown /proc/66960/task/68281/exe (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68281 ms_local ceph NOFD /proc/66960/task/68281/fd (opendir: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68282 ms_dispat ceph cwd unknown /proc/66960/task/68282/cwd (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68282 ms_dispat ceph rtd unknown /proc/66960/task/68282/root (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68282 ms_dispat ceph txt unknown /proc/66960/task/68282/exe (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68282 ms_dispat ceph NOFD /proc/66960/task/68282/fd (opendir: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68283 ms_local ceph cwd unknown /proc/66960/task/68283/cwd (readlink: Permission denied) 2024-03-16T12:24:12.392 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68283 ms_local ceph rtd unknown /proc/66960/task/68283/root (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68283 ms_local ceph txt unknown /proc/66960/task/68283/exe (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68283 ms_local ceph NOFD /proc/66960/task/68283/fd (opendir: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68284 ms_dispat ceph cwd unknown /proc/66960/task/68284/cwd (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68284 ms_dispat ceph rtd unknown /proc/66960/task/68284/root (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68284 ms_dispat ceph txt unknown /proc/66960/task/68284/exe (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68284 ms_dispat ceph NOFD /proc/66960/task/68284/fd (opendir: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68285 ms_local ceph cwd unknown /proc/66960/task/68285/cwd (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68285 ms_local ceph rtd unknown /proc/66960/task/68285/root (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68285 ms_local ceph txt unknown /proc/66960/task/68285/exe (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68285 ms_local ceph NOFD /proc/66960/task/68285/fd (opendir: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68286 ms_dispat ceph cwd unknown /proc/66960/task/68286/cwd (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68286 ms_dispat ceph rtd unknown /proc/66960/task/68286/root (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68286 ms_dispat ceph txt unknown /proc/66960/task/68286/exe (readlink: Permission denied) 2024-03-16T12:24:12.393 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68286 ms_dispat ceph NOFD /proc/66960/task/68286/fd (opendir: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68287 ms_local ceph cwd unknown /proc/66960/task/68287/cwd (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68287 ms_local ceph rtd unknown /proc/66960/task/68287/root (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68287 ms_local ceph txt unknown /proc/66960/task/68287/exe (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68287 ms_local ceph NOFD /proc/66960/task/68287/fd (opendir: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68288 ms_dispat ceph cwd unknown /proc/66960/task/68288/cwd (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68288 ms_dispat ceph rtd unknown /proc/66960/task/68288/root (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68288 ms_dispat ceph txt unknown /proc/66960/task/68288/exe (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68288 ms_dispat ceph NOFD /proc/66960/task/68288/fd (opendir: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68289 ms_local ceph cwd unknown /proc/66960/task/68289/cwd (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68289 ms_local ceph rtd unknown /proc/66960/task/68289/root (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68289 ms_local ceph txt unknown /proc/66960/task/68289/exe (readlink: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68289 ms_local ceph NOFD /proc/66960/task/68289/fd (opendir: Permission denied) 2024-03-16T12:24:12.394 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68290 fn_anonym ceph cwd unknown /proc/66960/task/68290/cwd (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68290 fn_anonym ceph rtd unknown /proc/66960/task/68290/root (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68290 fn_anonym ceph txt unknown /proc/66960/task/68290/exe (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68290 fn_anonym ceph NOFD /proc/66960/task/68290/fd (opendir: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68291 finisher ceph cwd unknown /proc/66960/task/68291/cwd (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68291 finisher ceph rtd unknown /proc/66960/task/68291/root (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68291 finisher ceph txt unknown /proc/66960/task/68291/exe (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68291 finisher ceph NOFD /proc/66960/task/68291/fd (opendir: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68292 safe_time ceph cwd unknown /proc/66960/task/68292/cwd (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68292 safe_time ceph rtd unknown /proc/66960/task/68292/root (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68292 safe_time ceph txt unknown /proc/66960/task/68292/exe (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68292 safe_time ceph NOFD /proc/66960/task/68292/fd (opendir: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68293 safe_time ceph cwd unknown /proc/66960/task/68293/cwd (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68293 safe_time ceph rtd unknown /proc/66960/task/68293/root (readlink: Permission denied) 2024-03-16T12:24:12.395 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68293 safe_time ceph txt unknown /proc/66960/task/68293/exe (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68293 safe_time ceph NOFD /proc/66960/task/68293/fd (opendir: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68294 ceph-osd ceph cwd unknown /proc/66960/task/68294/cwd (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68294 ceph-osd ceph rtd unknown /proc/66960/task/68294/root (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68294 ceph-osd ceph txt unknown /proc/66960/task/68294/exe (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68294 ceph-osd ceph NOFD /proc/66960/task/68294/fd (opendir: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68295 osd_srv_a ceph cwd unknown /proc/66960/task/68295/cwd (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68295 osd_srv_a ceph rtd unknown /proc/66960/task/68295/root (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68295 osd_srv_a ceph txt unknown /proc/66960/task/68295/exe (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68295 osd_srv_a ceph NOFD /proc/66960/task/68295/fd (opendir: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68296 tp_osd_tp ceph cwd unknown /proc/66960/task/68296/cwd (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68296 tp_osd_tp ceph rtd unknown /proc/66960/task/68296/root (readlink: Permission denied) 2024-03-16T12:24:12.396 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68296 tp_osd_tp ceph txt unknown /proc/66960/task/68296/exe (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68296 tp_osd_tp ceph NOFD /proc/66960/task/68296/fd (opendir: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68297 tp_osd_tp ceph cwd unknown /proc/66960/task/68297/cwd (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68297 tp_osd_tp ceph rtd unknown /proc/66960/task/68297/root (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68297 tp_osd_tp ceph txt unknown /proc/66960/task/68297/exe (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68297 tp_osd_tp ceph NOFD /proc/66960/task/68297/fd (opendir: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68298 tp_osd_tp ceph cwd unknown /proc/66960/task/68298/cwd (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68298 tp_osd_tp ceph rtd unknown /proc/66960/task/68298/root (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68298 tp_osd_tp ceph txt unknown /proc/66960/task/68298/exe (readlink: Permission denied) 2024-03-16T12:24:12.397 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68298 tp_osd_tp ceph NOFD /proc/66960/task/68298/fd (opendir: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68299 tp_osd_tp ceph cwd unknown /proc/66960/task/68299/cwd (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68299 tp_osd_tp ceph rtd unknown /proc/66960/task/68299/root (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68299 tp_osd_tp ceph txt unknown /proc/66960/task/68299/exe (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68299 tp_osd_tp ceph NOFD /proc/66960/task/68299/fd (opendir: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68300 tp_osd_tp ceph cwd unknown /proc/66960/task/68300/cwd (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68300 tp_osd_tp ceph rtd unknown /proc/66960/task/68300/root (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68300 tp_osd_tp ceph txt unknown /proc/66960/task/68300/exe (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68300 tp_osd_tp ceph NOFD /proc/66960/task/68300/fd (opendir: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68301 tp_osd_tp ceph cwd unknown /proc/66960/task/68301/cwd (readlink: Permission denied) 2024-03-16T12:24:12.398 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68301 tp_osd_tp ceph rtd unknown /proc/66960/task/68301/root (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68301 tp_osd_tp ceph txt unknown /proc/66960/task/68301/exe (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68301 tp_osd_tp ceph NOFD /proc/66960/task/68301/fd (opendir: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68302 tp_osd_tp ceph cwd unknown /proc/66960/task/68302/cwd (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68302 tp_osd_tp ceph rtd unknown /proc/66960/task/68302/root (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68302 tp_osd_tp ceph txt unknown /proc/66960/task/68302/exe (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68302 tp_osd_tp ceph NOFD /proc/66960/task/68302/fd (opendir: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68303 tp_osd_tp ceph cwd unknown /proc/66960/task/68303/cwd (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68303 tp_osd_tp ceph rtd unknown /proc/66960/task/68303/root (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68303 tp_osd_tp ceph txt unknown /proc/66960/task/68303/exe (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68303 tp_osd_tp ceph NOFD /proc/66960/task/68303/fd (opendir: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68304 tp_osd_tp ceph cwd unknown /proc/66960/task/68304/cwd (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68304 tp_osd_tp ceph rtd unknown /proc/66960/task/68304/root (readlink: Permission denied) 2024-03-16T12:24:12.399 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68304 tp_osd_tp ceph txt unknown /proc/66960/task/68304/exe (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68304 tp_osd_tp ceph NOFD /proc/66960/task/68304/fd (opendir: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68305 tp_osd_tp ceph cwd unknown /proc/66960/task/68305/cwd (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68305 tp_osd_tp ceph rtd unknown /proc/66960/task/68305/root (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68305 tp_osd_tp ceph txt unknown /proc/66960/task/68305/exe (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68305 tp_osd_tp ceph NOFD /proc/66960/task/68305/fd (opendir: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68306 tp_osd_tp ceph cwd unknown /proc/66960/task/68306/cwd (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68306 tp_osd_tp ceph rtd unknown /proc/66960/task/68306/root (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68306 tp_osd_tp ceph txt unknown /proc/66960/task/68306/exe (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68306 tp_osd_tp ceph NOFD /proc/66960/task/68306/fd (opendir: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68307 tp_osd_tp ceph cwd unknown /proc/66960/task/68307/cwd (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68307 tp_osd_tp ceph rtd unknown /proc/66960/task/68307/root (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68307 tp_osd_tp ceph txt unknown /proc/66960/task/68307/exe (readlink: Permission denied) 2024-03-16T12:24:12.400 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68307 tp_osd_tp ceph NOFD /proc/66960/task/68307/fd (opendir: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68308 tp_osd_tp ceph cwd unknown /proc/66960/task/68308/cwd (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68308 tp_osd_tp ceph rtd unknown /proc/66960/task/68308/root (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68308 tp_osd_tp ceph txt unknown /proc/66960/task/68308/exe (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68308 tp_osd_tp ceph NOFD /proc/66960/task/68308/fd (opendir: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68309 tp_osd_tp ceph cwd unknown /proc/66960/task/68309/cwd (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68309 tp_osd_tp ceph rtd unknown /proc/66960/task/68309/root (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68309 tp_osd_tp ceph txt unknown /proc/66960/task/68309/exe (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68309 tp_osd_tp ceph NOFD /proc/66960/task/68309/fd (opendir: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68310 tp_osd_tp ceph cwd unknown /proc/66960/task/68310/cwd (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68310 tp_osd_tp ceph rtd unknown /proc/66960/task/68310/root (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68310 tp_osd_tp ceph txt unknown /proc/66960/task/68310/exe (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68310 tp_osd_tp ceph NOFD /proc/66960/task/68310/fd (opendir: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68311 tp_osd_tp ceph cwd unknown /proc/66960/task/68311/cwd (readlink: Permission denied) 2024-03-16T12:24:12.401 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68311 tp_osd_tp ceph rtd unknown /proc/66960/task/68311/root (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68311 tp_osd_tp ceph txt unknown /proc/66960/task/68311/exe (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68311 tp_osd_tp ceph NOFD /proc/66960/task/68311/fd (opendir: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68312 osd_srv_h ceph cwd unknown /proc/66960/task/68312/cwd (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68312 osd_srv_h ceph rtd unknown /proc/66960/task/68312/root (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68312 osd_srv_h ceph txt unknown /proc/66960/task/68312/exe (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-osd 66960 68312 osd_srv_h ceph NOFD /proc/66960/task/68312/fd (opendir: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 68511 root cwd unknown /proc/68511/cwd (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 68511 root rtd unknown /proc/68511/root (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 68511 root txt unknown /proc/68511/exe (readlink: Permission denied) 2024-03-16T12:24:12.402 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 68511 root NOFD /proc/68511/fd (opendir: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 68542 root cwd unknown /proc/68542/cwd (readlink: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 68542 root rtd unknown /proc/68542/root (readlink: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 68542 root txt unknown /proc/68542/exe (readlink: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 68542 root NOFD /proc/68542/fd (opendir: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 71174 root cwd unknown /proc/71174/cwd (readlink: Permission denied) 2024-03-16T12:24:12.403 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 71174 root rtd unknown /proc/71174/root (readlink: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 71174 root txt unknown /proc/71174/exe (readlink: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 71174 root NOFD /proc/71174/fd (opendir: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71682 root cwd unknown /proc/71682/cwd (readlink: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71682 root rtd unknown /proc/71682/root (readlink: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71682 root txt unknown /proc/71682/exe (readlink: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71682 root NOFD /proc/71682/fd (opendir: Permission denied) 2024-03-16T12:24:12.404 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71699 root cwd unknown /proc/71699/cwd (readlink: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71699 root rtd unknown /proc/71699/root (readlink: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71699 root txt unknown /proc/71699/exe (readlink: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71699 root NOFD /proc/71699/fd (opendir: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71706 root cwd unknown /proc/71706/cwd (readlink: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71706 root rtd unknown /proc/71706/root (readlink: Permission denied) 2024-03-16T12:24:12.405 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71706 root txt unknown /proc/71706/exe (readlink: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71706 root NOFD /proc/71706/fd (opendir: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71719 root cwd unknown /proc/71719/cwd (readlink: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71719 root rtd unknown /proc/71719/root (readlink: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71719 root txt unknown /proc/71719/exe (readlink: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71719 root NOFD /proc/71719/fd (opendir: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71728 root cwd unknown /proc/71728/cwd (readlink: Permission denied) 2024-03-16T12:24:12.406 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71728 root rtd unknown /proc/71728/root (readlink: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71728 root txt unknown /proc/71728/exe (readlink: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71728 root NOFD /proc/71728/fd (opendir: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71737 root cwd unknown /proc/71737/cwd (readlink: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71737 root rtd unknown /proc/71737/root (readlink: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71737 root txt unknown /proc/71737/exe (readlink: Permission denied) 2024-03-16T12:24:12.407 INFO:tasks.workunit.client.0.smithi098.stdout:sudo 71737 root NOFD /proc/71737/fd (opendir: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71748 root cwd unknown /proc/71748/cwd (readlink: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71748 root rtd unknown /proc/71748/root (readlink: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71748 root txt unknown /proc/71748/exe (readlink: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71748 root NOFD /proc/71748/fd (opendir: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71758 root cwd unknown /proc/71758/cwd (readlink: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71758 root rtd unknown /proc/71758/root (readlink: Permission denied) 2024-03-16T12:24:12.408 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71758 root txt unknown /proc/71758/exe (readlink: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:journalct 71758 root NOFD /proc/71758/fd (opendir: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 71894 root cwd unknown /proc/71894/cwd (readlink: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 71894 root rtd unknown /proc/71894/root (readlink: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 71894 root txt unknown /proc/71894/exe (readlink: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 71894 root NOFD /proc/71894/fd (opendir: Permission denied) 2024-03-16T12:24:12.409 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72290 root cwd unknown /proc/72290/cwd (readlink: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72290 root rtd unknown /proc/72290/root (readlink: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72290 root txt unknown /proc/72290/exe (readlink: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72290 root NOFD /proc/72290/fd (opendir: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72292 root cwd unknown /proc/72292/cwd (readlink: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72292 root rtd unknown /proc/72292/root (readlink: Permission denied) 2024-03-16T12:24:12.410 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72292 root txt unknown /proc/72292/exe (readlink: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72292 root NOFD /proc/72292/fd (opendir: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 ceph cwd unknown /proc/72294/cwd (readlink: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 ceph rtd unknown /proc/72294/root (readlink: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 ceph txt unknown /proc/72294/exe (readlink: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 ceph NOFD /proc/72294/fd (opendir: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72295 log ceph cwd unknown /proc/72294/task/72295/cwd (readlink: Permission denied) 2024-03-16T12:24:12.411 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72295 log ceph rtd unknown /proc/72294/task/72295/root (readlink: Permission denied) 2024-03-16T12:24:12.412 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72295 log ceph txt unknown /proc/72294/task/72295/exe (readlink: Permission denied) 2024-03-16T12:24:12.412 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72295 log ceph NOFD /proc/72294/task/72295/fd (opendir: Permission denied) 2024-03-16T12:24:12.412 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72297 msgr-work ceph cwd unknown /proc/72294/task/72297/cwd (readlink: Permission denied) 2024-03-16T12:24:12.412 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72297 msgr-work ceph rtd unknown /proc/72294/task/72297/root (readlink: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72297 msgr-work ceph txt unknown /proc/72294/task/72297/exe (readlink: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72297 msgr-work ceph NOFD /proc/72294/task/72297/fd (opendir: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72298 msgr-work ceph cwd unknown /proc/72294/task/72298/cwd (readlink: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72298 msgr-work ceph rtd unknown /proc/72294/task/72298/root (readlink: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72298 msgr-work ceph txt unknown /proc/72294/task/72298/exe (readlink: Permission denied) 2024-03-16T12:24:12.413 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72298 msgr-work ceph NOFD /proc/72294/task/72298/fd (opendir: Permission denied) 2024-03-16T12:24:12.414 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72299 msgr-work ceph cwd unknown /proc/72294/task/72299/cwd (readlink: Permission denied) 2024-03-16T12:24:12.414 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72299 msgr-work ceph rtd unknown /proc/72294/task/72299/root (readlink: Permission denied) 2024-03-16T12:24:12.414 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72299 msgr-work ceph txt unknown /proc/72294/task/72299/exe (readlink: Permission denied) 2024-03-16T12:24:12.414 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72299 msgr-work ceph NOFD /proc/72294/task/72299/fd (opendir: Permission denied) 2024-03-16T12:24:12.414 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72303 service ceph cwd unknown /proc/72294/task/72303/cwd (readlink: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72303 service ceph rtd unknown /proc/72294/task/72303/root (readlink: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72303 service ceph txt unknown /proc/72294/task/72303/exe (readlink: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72303 service ceph NOFD /proc/72294/task/72303/fd (opendir: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72304 admin_soc ceph cwd unknown /proc/72294/task/72304/cwd (readlink: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72304 admin_soc ceph rtd unknown /proc/72294/task/72304/root (readlink: Permission denied) 2024-03-16T12:24:12.415 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72304 admin_soc ceph txt unknown /proc/72294/task/72304/exe (readlink: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72304 admin_soc ceph NOFD /proc/72294/task/72304/fd (opendir: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72305 signal_ha ceph cwd unknown /proc/72294/task/72305/cwd (readlink: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72305 signal_ha ceph rtd unknown /proc/72294/task/72305/root (readlink: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72305 signal_ha ceph txt unknown /proc/72294/task/72305/exe (readlink: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72305 signal_ha ceph NOFD /proc/72294/task/72305/fd (opendir: Permission denied) 2024-03-16T12:24:12.416 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72306 io_contex ceph cwd unknown /proc/72294/task/72306/cwd (readlink: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72306 io_contex ceph rtd unknown /proc/72294/task/72306/root (readlink: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72306 io_contex ceph txt unknown /proc/72294/task/72306/exe (readlink: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72306 io_contex ceph NOFD /proc/72294/task/72306/fd (opendir: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72307 io_contex ceph cwd unknown /proc/72294/task/72307/cwd (readlink: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72307 io_contex ceph rtd unknown /proc/72294/task/72307/root (readlink: Permission denied) 2024-03-16T12:24:12.417 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72307 io_contex ceph txt unknown /proc/72294/task/72307/exe (readlink: Permission denied) 2024-03-16T12:24:12.418 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72307 io_contex ceph NOFD /proc/72294/task/72307/fd (opendir: Permission denied) 2024-03-16T12:24:12.418 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72308 ms_dispat ceph cwd unknown /proc/72294/task/72308/cwd (readlink: Permission denied) 2024-03-16T12:24:12.418 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72308 ms_dispat ceph rtd unknown /proc/72294/task/72308/root (readlink: Permission denied) 2024-03-16T12:24:12.418 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72308 ms_dispat ceph txt unknown /proc/72294/task/72308/exe (readlink: Permission denied) 2024-03-16T12:24:12.418 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72308 ms_dispat ceph NOFD /proc/72294/task/72308/fd (opendir: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72309 ms_local ceph cwd unknown /proc/72294/task/72309/cwd (readlink: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72309 ms_local ceph rtd unknown /proc/72294/task/72309/root (readlink: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72309 ms_local ceph txt unknown /proc/72294/task/72309/exe (readlink: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72309 ms_local ceph NOFD /proc/72294/task/72309/fd (opendir: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72310 safe_time ceph cwd unknown /proc/72294/task/72310/cwd (readlink: Permission denied) 2024-03-16T12:24:12.419 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72310 safe_time ceph rtd unknown /proc/72294/task/72310/root (readlink: Permission denied) 2024-03-16T12:24:12.420 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72310 safe_time ceph txt unknown /proc/72294/task/72310/exe (readlink: Permission denied) 2024-03-16T12:24:12.420 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72310 safe_time ceph NOFD /proc/72294/task/72310/fd (opendir: Permission denied) 2024-03-16T12:24:12.420 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72311 safe_time ceph cwd unknown /proc/72294/task/72311/cwd (readlink: Permission denied) 2024-03-16T12:24:12.420 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72311 safe_time ceph rtd unknown /proc/72294/task/72311/root (readlink: Permission denied) 2024-03-16T12:24:12.421 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72311 safe_time ceph txt unknown /proc/72294/task/72311/exe (readlink: Permission denied) 2024-03-16T12:24:12.421 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72311 safe_time ceph NOFD /proc/72294/task/72311/fd (opendir: Permission denied) 2024-03-16T12:24:12.421 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72312 ceph-mds ceph cwd unknown /proc/72294/task/72312/cwd (readlink: Permission denied) 2024-03-16T12:24:12.421 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72312 ceph-mds ceph rtd unknown /proc/72294/task/72312/root (readlink: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72312 ceph-mds ceph txt unknown /proc/72294/task/72312/exe (readlink: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72312 ceph-mds ceph NOFD /proc/72294/task/72312/fd (opendir: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72341 safe_time ceph cwd unknown /proc/72294/task/72341/cwd (readlink: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72341 safe_time ceph rtd unknown /proc/72294/task/72341/root (readlink: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72341 safe_time ceph txt unknown /proc/72294/task/72341/exe (readlink: Permission denied) 2024-03-16T12:24:12.422 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72294 72341 safe_time ceph NOFD /proc/72294/task/72341/fd (opendir: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72639 root cwd unknown /proc/72639/cwd (readlink: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72639 root rtd unknown /proc/72639/root (readlink: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72639 root txt unknown /proc/72639/exe (readlink: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 72639 root NOFD /proc/72639/fd (opendir: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72641 root cwd unknown /proc/72641/cwd (readlink: Permission denied) 2024-03-16T12:24:12.423 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72641 root rtd unknown /proc/72641/root (readlink: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72641 root txt unknown /proc/72641/exe (readlink: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 72641 root NOFD /proc/72641/fd (opendir: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 ceph cwd unknown /proc/72663/cwd (readlink: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 ceph rtd unknown /proc/72663/root (readlink: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 ceph txt unknown /proc/72663/exe (readlink: Permission denied) 2024-03-16T12:24:12.424 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 ceph NOFD /proc/72663/fd (opendir: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72664 log ceph cwd unknown /proc/72663/task/72664/cwd (readlink: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72664 log ceph rtd unknown /proc/72663/task/72664/root (readlink: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72664 log ceph txt unknown /proc/72663/task/72664/exe (readlink: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72664 log ceph NOFD /proc/72663/task/72664/fd (opendir: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72666 msgr-work ceph cwd unknown /proc/72663/task/72666/cwd (readlink: Permission denied) 2024-03-16T12:24:12.425 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72666 msgr-work ceph rtd unknown /proc/72663/task/72666/root (readlink: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72666 msgr-work ceph txt unknown /proc/72663/task/72666/exe (readlink: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72666 msgr-work ceph NOFD /proc/72663/task/72666/fd (opendir: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72667 msgr-work ceph cwd unknown /proc/72663/task/72667/cwd (readlink: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72667 msgr-work ceph rtd unknown /proc/72663/task/72667/root (readlink: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72667 msgr-work ceph txt unknown /proc/72663/task/72667/exe (readlink: Permission denied) 2024-03-16T12:24:12.426 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72667 msgr-work ceph NOFD /proc/72663/task/72667/fd (opendir: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72668 msgr-work ceph cwd unknown /proc/72663/task/72668/cwd (readlink: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72668 msgr-work ceph rtd unknown /proc/72663/task/72668/root (readlink: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72668 msgr-work ceph txt unknown /proc/72663/task/72668/exe (readlink: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72668 msgr-work ceph NOFD /proc/72663/task/72668/fd (opendir: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72672 service ceph cwd unknown /proc/72663/task/72672/cwd (readlink: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72672 service ceph rtd unknown /proc/72663/task/72672/root (readlink: Permission denied) 2024-03-16T12:24:12.427 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72672 service ceph txt unknown /proc/72663/task/72672/exe (readlink: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72672 service ceph NOFD /proc/72663/task/72672/fd (opendir: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72673 admin_soc ceph cwd unknown /proc/72663/task/72673/cwd (readlink: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72673 admin_soc ceph rtd unknown /proc/72663/task/72673/root (readlink: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72673 admin_soc ceph txt unknown /proc/72663/task/72673/exe (readlink: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72673 admin_soc ceph NOFD /proc/72663/task/72673/fd (opendir: Permission denied) 2024-03-16T12:24:12.428 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72674 signal_ha ceph cwd unknown /proc/72663/task/72674/cwd (readlink: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72674 signal_ha ceph rtd unknown /proc/72663/task/72674/root (readlink: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72674 signal_ha ceph txt unknown /proc/72663/task/72674/exe (readlink: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72674 signal_ha ceph NOFD /proc/72663/task/72674/fd (opendir: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72675 io_contex ceph cwd unknown /proc/72663/task/72675/cwd (readlink: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72675 io_contex ceph rtd unknown /proc/72663/task/72675/root (readlink: Permission denied) 2024-03-16T12:24:12.429 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72675 io_contex ceph txt unknown /proc/72663/task/72675/exe (readlink: Permission denied) 2024-03-16T12:24:12.430 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72675 io_contex ceph NOFD /proc/72663/task/72675/fd (opendir: Permission denied) 2024-03-16T12:24:12.430 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72676 io_contex ceph cwd unknown /proc/72663/task/72676/cwd (readlink: Permission denied) 2024-03-16T12:24:12.430 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72676 io_contex ceph rtd unknown /proc/72663/task/72676/root (readlink: Permission denied) 2024-03-16T12:24:12.430 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72676 io_contex ceph txt unknown /proc/72663/task/72676/exe (readlink: Permission denied) 2024-03-16T12:24:12.430 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72676 io_contex ceph NOFD /proc/72663/task/72676/fd (opendir: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72677 ms_dispat ceph cwd unknown /proc/72663/task/72677/cwd (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72677 ms_dispat ceph rtd unknown /proc/72663/task/72677/root (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72677 ms_dispat ceph txt unknown /proc/72663/task/72677/exe (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72677 ms_dispat ceph NOFD /proc/72663/task/72677/fd (opendir: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72678 ms_local ceph cwd unknown /proc/72663/task/72678/cwd (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72678 ms_local ceph rtd unknown /proc/72663/task/72678/root (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72678 ms_local ceph txt unknown /proc/72663/task/72678/exe (readlink: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72678 ms_local ceph NOFD /proc/72663/task/72678/fd (opendir: Permission denied) 2024-03-16T12:24:12.431 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72679 safe_time ceph cwd unknown /proc/72663/task/72679/cwd (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72679 safe_time ceph rtd unknown /proc/72663/task/72679/root (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72679 safe_time ceph txt unknown /proc/72663/task/72679/exe (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72679 safe_time ceph NOFD /proc/72663/task/72679/fd (opendir: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72680 safe_time ceph cwd unknown /proc/72663/task/72680/cwd (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72680 safe_time ceph rtd unknown /proc/72663/task/72680/root (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72680 safe_time ceph txt unknown /proc/72663/task/72680/exe (readlink: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72680 safe_time ceph NOFD /proc/72663/task/72680/fd (opendir: Permission denied) 2024-03-16T12:24:12.432 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72681 ceph-mds ceph cwd unknown /proc/72663/task/72681/cwd (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72681 ceph-mds ceph rtd unknown /proc/72663/task/72681/root (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72681 ceph-mds ceph txt unknown /proc/72663/task/72681/exe (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72681 ceph-mds ceph NOFD /proc/72663/task/72681/fd (opendir: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72705 safe_time ceph cwd unknown /proc/72663/task/72705/cwd (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72705 safe_time ceph rtd unknown /proc/72663/task/72705/root (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72705 safe_time ceph txt unknown /proc/72663/task/72705/exe (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 72663 72705 safe_time ceph NOFD /proc/72663/task/72705/fd (opendir: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73277 root cwd unknown /proc/73277/cwd (readlink: Permission denied) 2024-03-16T12:24:12.433 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73277 root rtd unknown /proc/73277/root (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73277 root txt unknown /proc/73277/exe (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73277 root NOFD /proc/73277/fd (opendir: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73279 root cwd unknown /proc/73279/cwd (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73279 root rtd unknown /proc/73279/root (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73279 root txt unknown /proc/73279/exe (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73279 root NOFD /proc/73279/fd (opendir: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 ceph cwd unknown /proc/73281/cwd (readlink: Permission denied) 2024-03-16T12:24:12.434 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 ceph rtd unknown /proc/73281/root (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 ceph txt unknown /proc/73281/exe (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 ceph NOFD /proc/73281/fd (opendir: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73282 log ceph cwd unknown /proc/73281/task/73282/cwd (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73282 log ceph rtd unknown /proc/73281/task/73282/root (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73282 log ceph txt unknown /proc/73281/task/73282/exe (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73282 log ceph NOFD /proc/73281/task/73282/fd (opendir: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73284 msgr-work ceph cwd unknown /proc/73281/task/73284/cwd (readlink: Permission denied) 2024-03-16T12:24:12.435 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73284 msgr-work ceph rtd unknown /proc/73281/task/73284/root (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73284 msgr-work ceph txt unknown /proc/73281/task/73284/exe (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73284 msgr-work ceph NOFD /proc/73281/task/73284/fd (opendir: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73285 msgr-work ceph cwd unknown /proc/73281/task/73285/cwd (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73285 msgr-work ceph rtd unknown /proc/73281/task/73285/root (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73285 msgr-work ceph txt unknown /proc/73281/task/73285/exe (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73285 msgr-work ceph NOFD /proc/73281/task/73285/fd (opendir: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73286 msgr-work ceph cwd unknown /proc/73281/task/73286/cwd (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73286 msgr-work ceph rtd unknown /proc/73281/task/73286/root (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73286 msgr-work ceph txt unknown /proc/73281/task/73286/exe (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73286 msgr-work ceph NOFD /proc/73281/task/73286/fd (opendir: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73290 service ceph cwd unknown /proc/73281/task/73290/cwd (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73290 service ceph rtd unknown /proc/73281/task/73290/root (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73290 service ceph txt unknown /proc/73281/task/73290/exe (readlink: Permission denied) 2024-03-16T12:24:12.436 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73290 service ceph NOFD /proc/73281/task/73290/fd (opendir: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73291 admin_soc ceph cwd unknown /proc/73281/task/73291/cwd (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73291 admin_soc ceph rtd unknown /proc/73281/task/73291/root (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73291 admin_soc ceph txt unknown /proc/73281/task/73291/exe (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73291 admin_soc ceph NOFD /proc/73281/task/73291/fd (opendir: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73292 signal_ha ceph cwd unknown /proc/73281/task/73292/cwd (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73292 signal_ha ceph rtd unknown /proc/73281/task/73292/root (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73292 signal_ha ceph txt unknown /proc/73281/task/73292/exe (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73292 signal_ha ceph NOFD /proc/73281/task/73292/fd (opendir: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73293 io_contex ceph cwd unknown /proc/73281/task/73293/cwd (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73293 io_contex ceph rtd unknown /proc/73281/task/73293/root (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73293 io_contex ceph txt unknown /proc/73281/task/73293/exe (readlink: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73293 io_contex ceph NOFD /proc/73281/task/73293/fd (opendir: Permission denied) 2024-03-16T12:24:12.437 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73294 io_contex ceph cwd unknown /proc/73281/task/73294/cwd (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73294 io_contex ceph rtd unknown /proc/73281/task/73294/root (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73294 io_contex ceph txt unknown /proc/73281/task/73294/exe (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73294 io_contex ceph NOFD /proc/73281/task/73294/fd (opendir: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73295 ms_dispat ceph cwd unknown /proc/73281/task/73295/cwd (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73295 ms_dispat ceph rtd unknown /proc/73281/task/73295/root (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73295 ms_dispat ceph txt unknown /proc/73281/task/73295/exe (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73295 ms_dispat ceph NOFD /proc/73281/task/73295/fd (opendir: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73296 ms_local ceph cwd unknown /proc/73281/task/73296/cwd (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73296 ms_local ceph rtd unknown /proc/73281/task/73296/root (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73296 ms_local ceph txt unknown /proc/73281/task/73296/exe (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73296 ms_local ceph NOFD /proc/73281/task/73296/fd (opendir: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73297 safe_time ceph cwd unknown /proc/73281/task/73297/cwd (readlink: Permission denied) 2024-03-16T12:24:12.438 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73297 safe_time ceph rtd unknown /proc/73281/task/73297/root (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73297 safe_time ceph txt unknown /proc/73281/task/73297/exe (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73297 safe_time ceph NOFD /proc/73281/task/73297/fd (opendir: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73298 safe_time ceph cwd unknown /proc/73281/task/73298/cwd (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73298 safe_time ceph rtd unknown /proc/73281/task/73298/root (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73298 safe_time ceph txt unknown /proc/73281/task/73298/exe (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73298 safe_time ceph NOFD /proc/73281/task/73298/fd (opendir: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73299 ceph-mds ceph cwd unknown /proc/73281/task/73299/cwd (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73299 ceph-mds ceph rtd unknown /proc/73281/task/73299/root (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73299 ceph-mds ceph txt unknown /proc/73281/task/73299/exe (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73299 ceph-mds ceph NOFD /proc/73281/task/73299/fd (opendir: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73371 safe_time ceph cwd unknown /proc/73281/task/73371/cwd (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73371 safe_time ceph rtd unknown /proc/73281/task/73371/root (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73371 safe_time ceph txt unknown /proc/73281/task/73371/exe (readlink: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73281 73371 safe_time ceph NOFD /proc/73281/task/73371/fd (opendir: Permission denied) 2024-03-16T12:24:12.439 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 73304 root cwd unknown /proc/73304/cwd (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 73304 root rtd unknown /proc/73304/root (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 73304 root txt unknown /proc/73304/exe (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/6 73304 root NOFD /proc/73304/fd (opendir: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73644 root cwd unknown /proc/73644/cwd (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73644 root rtd unknown /proc/73644/root (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73644 root txt unknown /proc/73644/exe (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:conmon 73644 root NOFD /proc/73644/fd (opendir: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73646 root cwd unknown /proc/73646/cwd (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73646 root rtd unknown /proc/73646/root (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73646 root txt unknown /proc/73646/exe (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:podman-in 73646 root NOFD /proc/73646/fd (opendir: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 ceph cwd unknown /proc/73667/cwd (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 ceph rtd unknown /proc/73667/root (readlink: Permission denied) 2024-03-16T12:24:12.440 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 ceph txt unknown /proc/73667/exe (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 ceph NOFD /proc/73667/fd (opendir: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73668 log ceph cwd unknown /proc/73667/task/73668/cwd (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73668 log ceph rtd unknown /proc/73667/task/73668/root (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73668 log ceph txt unknown /proc/73667/task/73668/exe (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73668 log ceph NOFD /proc/73667/task/73668/fd (opendir: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73670 msgr-work ceph cwd unknown /proc/73667/task/73670/cwd (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73670 msgr-work ceph rtd unknown /proc/73667/task/73670/root (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73670 msgr-work ceph txt unknown /proc/73667/task/73670/exe (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73670 msgr-work ceph NOFD /proc/73667/task/73670/fd (opendir: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73671 msgr-work ceph cwd unknown /proc/73667/task/73671/cwd (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73671 msgr-work ceph rtd unknown /proc/73667/task/73671/root (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73671 msgr-work ceph txt unknown /proc/73667/task/73671/exe (readlink: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73671 msgr-work ceph NOFD /proc/73667/task/73671/fd (opendir: Permission denied) 2024-03-16T12:24:12.441 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73672 msgr-work ceph cwd unknown /proc/73667/task/73672/cwd (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73672 msgr-work ceph rtd unknown /proc/73667/task/73672/root (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73672 msgr-work ceph txt unknown /proc/73667/task/73672/exe (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73672 msgr-work ceph NOFD /proc/73667/task/73672/fd (opendir: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73676 service ceph cwd unknown /proc/73667/task/73676/cwd (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73676 service ceph rtd unknown /proc/73667/task/73676/root (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73676 service ceph txt unknown /proc/73667/task/73676/exe (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73676 service ceph NOFD /proc/73667/task/73676/fd (opendir: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73677 admin_soc ceph cwd unknown /proc/73667/task/73677/cwd (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73677 admin_soc ceph rtd unknown /proc/73667/task/73677/root (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73677 admin_soc ceph txt unknown /proc/73667/task/73677/exe (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73677 admin_soc ceph NOFD /proc/73667/task/73677/fd (opendir: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73678 signal_ha ceph cwd unknown /proc/73667/task/73678/cwd (readlink: Permission denied) 2024-03-16T12:24:12.442 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73678 signal_ha ceph rtd unknown /proc/73667/task/73678/root (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73678 signal_ha ceph txt unknown /proc/73667/task/73678/exe (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73678 signal_ha ceph NOFD /proc/73667/task/73678/fd (opendir: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73679 io_contex ceph cwd unknown /proc/73667/task/73679/cwd (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73679 io_contex ceph rtd unknown /proc/73667/task/73679/root (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73679 io_contex ceph txt unknown /proc/73667/task/73679/exe (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73679 io_contex ceph NOFD /proc/73667/task/73679/fd (opendir: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73680 io_contex ceph cwd unknown /proc/73667/task/73680/cwd (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73680 io_contex ceph rtd unknown /proc/73667/task/73680/root (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73680 io_contex ceph txt unknown /proc/73667/task/73680/exe (readlink: Permission denied) 2024-03-16T12:24:12.443 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73680 io_contex ceph NOFD /proc/73667/task/73680/fd (opendir: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73681 ms_dispat ceph cwd unknown /proc/73667/task/73681/cwd (readlink: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73681 ms_dispat ceph rtd unknown /proc/73667/task/73681/root (readlink: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73681 ms_dispat ceph txt unknown /proc/73667/task/73681/exe (readlink: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73681 ms_dispat ceph NOFD /proc/73667/task/73681/fd (opendir: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73682 ms_local ceph cwd unknown /proc/73667/task/73682/cwd (readlink: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73682 ms_local ceph rtd unknown /proc/73667/task/73682/root (readlink: Permission denied) 2024-03-16T12:24:12.444 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73682 ms_local ceph txt unknown /proc/73667/task/73682/exe (readlink: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73682 ms_local ceph NOFD /proc/73667/task/73682/fd (opendir: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73683 safe_time ceph cwd unknown /proc/73667/task/73683/cwd (readlink: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73683 safe_time ceph rtd unknown /proc/73667/task/73683/root (readlink: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73683 safe_time ceph txt unknown /proc/73667/task/73683/exe (readlink: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73683 safe_time ceph NOFD /proc/73667/task/73683/fd (opendir: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73684 safe_time ceph cwd unknown /proc/73667/task/73684/cwd (readlink: Permission denied) 2024-03-16T12:24:12.445 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73684 safe_time ceph rtd unknown /proc/73667/task/73684/root (readlink: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73684 safe_time ceph txt unknown /proc/73667/task/73684/exe (readlink: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73684 safe_time ceph NOFD /proc/73667/task/73684/fd (opendir: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73685 ceph-mds ceph cwd unknown /proc/73667/task/73685/cwd (readlink: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73685 ceph-mds ceph rtd unknown /proc/73667/task/73685/root (readlink: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73685 ceph-mds ceph txt unknown /proc/73667/task/73685/exe (readlink: Permission denied) 2024-03-16T12:24:12.446 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73685 ceph-mds ceph NOFD /proc/73667/task/73685/fd (opendir: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73692 safe_time ceph cwd unknown /proc/73667/task/73692/cwd (readlink: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73692 safe_time ceph rtd unknown /proc/73667/task/73692/root (readlink: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73692 safe_time ceph txt unknown /proc/73667/task/73692/exe (readlink: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:ceph-mds 73667 73692 safe_time ceph NOFD /proc/73667/task/73692/fd (opendir: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 75381 root cwd unknown /proc/75381/cwd (readlink: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 75381 root rtd unknown /proc/75381/root (readlink: Permission denied) 2024-03-16T12:24:12.447 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 75381 root txt unknown /proc/75381/exe (readlink: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 75381 root NOFD /proc/75381/fd (opendir: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 78869 root cwd unknown /proc/78869/cwd (readlink: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 78869 root rtd unknown /proc/78869/root (readlink: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 78869 root txt unknown /proc/78869/exe (readlink: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 78869 root NOFD /proc/78869/fd (opendir: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 79584 root cwd unknown /proc/79584/cwd (readlink: Permission denied) 2024-03-16T12:24:12.448 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 79584 root rtd unknown /proc/79584/root (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 79584 root txt unknown /proc/79584/exe (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/u 79584 root NOFD /proc/79584/fd (opendir: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 79956 root cwd unknown /proc/79956/cwd (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 79956 root rtd unknown /proc/79956/root (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 79956 root txt unknown /proc/79956/exe (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/5 79956 root NOFD /proc/79956/fd (opendir: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 80351 root cwd unknown /proc/80351/cwd (readlink: Permission denied) 2024-03-16T12:24:12.449 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 80351 root rtd unknown /proc/80351/root (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 80351 root txt unknown /proc/80351/exe (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/2 80351 root NOFD /proc/80351/fd (opendir: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81775 root cwd unknown /proc/81775/cwd (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81775 root rtd unknown /proc/81775/root (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81775 root txt unknown /proc/81775/exe (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81775 root NOFD /proc/81775/fd (opendir: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81816 root cwd unknown /proc/81816/cwd (readlink: Permission denied) 2024-03-16T12:24:12.450 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81816 root rtd unknown /proc/81816/root (readlink: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81816 root txt unknown /proc/81816/exe (readlink: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/3 81816 root NOFD /proc/81816/fd (opendir: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86085 root cwd unknown /proc/86085/cwd (readlink: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86085 root rtd unknown /proc/86085/root (readlink: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86085 root txt unknown /proc/86085/exe (readlink: Permission denied) 2024-03-16T12:24:12.451 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86085 root NOFD /proc/86085/fd (opendir: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86100 root cwd unknown /proc/86100/cwd (readlink: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86100 root rtd unknown /proc/86100/root (readlink: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86100 root txt unknown /proc/86100/exe (readlink: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86100 root NOFD /proc/86100/fd (opendir: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86101 root cwd unknown /proc/86101/cwd (readlink: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86101 root rtd unknown /proc/86101/root (readlink: Permission denied) 2024-03-16T12:24:12.452 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86101 root txt unknown /proc/86101/exe (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/R 86101 root NOFD /proc/86101/fd (opendir: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 86636 root cwd unknown /proc/86636/cwd (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 86636 root rtd unknown /proc/86636/root (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 86636 root txt unknown /proc/86636/exe (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/1 86636 root NOFD /proc/86636/fd (opendir: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88322 root cwd unknown /proc/88322/cwd (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88322 root rtd unknown /proc/88322/root (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88322 root txt unknown /proc/88322/exe (readlink: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88322 root NOFD /proc/88322/fd (opendir: Permission denied) 2024-03-16T12:24:12.453 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88323 root cwd unknown /proc/88323/cwd (readlink: Permission denied) 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88323 root rtd unknown /proc/88323/root (readlink: Permission denied) 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88323 root txt unknown /proc/88323/exe (readlink: Permission denied) 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/7 88323 root NOFD /proc/88323/fd (opendir: Permission denied) 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu txt REG 8,1 40944 10098 /usr/bin/timeout 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-16T12:24:12.454 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu 0r FIFO 0,13 0t0 214953 pipe 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu 1w FIFO 0,13 0t0 214954 pipe 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:timeout 88424 ubuntu 2w FIFO 0,13 0t0 214955 pipe 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-16T12:24:12.455 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T12:24:12.456 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu 0r FIFO 0,13 0t0 214953 pipe 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu 1w FIFO 0,13 0t0 214954 pipe 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu 2w FIFO 0,13 0t0 214955 pipe 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:fsync-tes 88442 ubuntu 255r REG 8,1 207 540808 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 89012 root cwd unknown /proc/89012/cwd (readlink: Permission denied) 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 89012 root rtd unknown /proc/89012/root (readlink: Permission denied) 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 89012 root txt unknown /proc/89012/exe (readlink: Permission denied) 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:kworker/0 89012 root NOFD /proc/89012/fd (opendir: Permission denied) 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T12:24:12.457 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T12:24:12.458 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 0r FIFO 0,13 0t0 214953 pipe 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 1w FIFO 0,13 0t0 214954 pipe 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 2w FIFO 0,13 0t0 214955 pipe 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 3r DIR 0,20 0 1 /proc 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 4r DIR 0,20 7 219450 /proc/89445/fd 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 5w FIFO 0,13 0t0 219455 pipe 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89445 ubuntu 6r FIFO 0,13 0t0 219456 pipe 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu rtd DIR 8,1 4096 2 / 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-16T12:24:12.459 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu 4r FIFO 0,13 0t0 219455 pipe 2024-03-16T12:24:12.460 INFO:tasks.workunit.client.0.smithi098.stdout:lsof 89446 ubuntu 7w FIFO 0,13 0t0 219456 pipe 2024-03-16T12:24:12.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:12 smithi173 ceph-mon[36632]: pgmap v485: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:24:12.462 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T12:24:12.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T12:24:12.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:12 smithi098 ceph-mon[31633]: pgmap v485: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:24:12.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:12 smithi153 ceph-mon[36901]: pgmap v485: 129 pgs: 129 active+clean; 258 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-03-16T12:24:12.589 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-03-16T12:24:12.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T12:24:13.634 DEBUG:teuthology.parallel:result is None 2024-03-16T12:24:13.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T12:24:13.802 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T12:24:13.802 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-16T12:24:13.815 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-16T12:24:14.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:14 smithi173 ceph-mon[36632]: pgmap v486: 129 pgs: 129 active+clean; 258 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 776 KiB/s wr, 9 op/s 2024-03-16T12:24:14.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:14 smithi098 ceph-mon[31633]: pgmap v486: 129 pgs: 129 active+clean; 258 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 776 KiB/s wr, 9 op/s 2024-03-16T12:24:14.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:14 smithi153 ceph-mon[36901]: pgmap v486: 129 pgs: 129 active+clean; 258 MiB data, 891 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 776 KiB/s wr, 9 op/s 2024-03-16T12:24:16.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:16 smithi098 ceph-mon[31633]: pgmap v487: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 MiB/s wr, 10 op/s 2024-03-16T12:24:16.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:16 smithi173 ceph-mon[36632]: pgmap v487: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 MiB/s wr, 10 op/s 2024-03-16T12:24:16.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:16 smithi153 ceph-mon[36901]: pgmap v487: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.8 MiB/s wr, 10 op/s 2024-03-16T12:24:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:18 smithi173 ceph-mon[36632]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 14 op/s 2024-03-16T12:24:18.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:18 smithi098 ceph-mon[31633]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 14 op/s 2024-03-16T12:24:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:18 smithi153 ceph-mon[36901]: pgmap v488: 129 pgs: 129 active+clean; 258 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 14 op/s 2024-03-16T12:24:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:20 smithi173 ceph-mon[36632]: pgmap v489: 129 pgs: 129 active+clean; 190 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 22 MiB/s wr, 15 op/s 2024-03-16T12:24:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:20 smithi098 ceph-mon[31633]: pgmap v489: 129 pgs: 129 active+clean; 190 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 22 MiB/s wr, 15 op/s 2024-03-16T12:24:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:20 smithi153 ceph-mon[36901]: pgmap v489: 129 pgs: 129 active+clean; 190 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 22 MiB/s wr, 15 op/s 2024-03-16T12:24:22.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:22 smithi173 ceph-mon[36632]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-03-16T12:24:22.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:22 smithi098 ceph-mon[31633]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-03-16T12:24:22.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:22 smithi153 ceph-mon[36901]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-03-16T12:24:23.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:23 smithi173 ceph-mon[36632]: pgmap v491: 129 pgs: 129 active+clean; 1.1 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 20 op/s 2024-03-16T12:24:23.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:23 smithi098 ceph-mon[31633]: pgmap v491: 129 pgs: 129 active+clean; 1.1 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 20 op/s 2024-03-16T12:24:23.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:23 smithi153 ceph-mon[36901]: pgmap v491: 129 pgs: 129 active+clean; 1.1 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 20 op/s 2024-03-16T12:24:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:25 smithi098 ceph-mon[31633]: pgmap v492: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 MiB/s wr, 19 op/s 2024-03-16T12:24:26.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:25 smithi153 ceph-mon[36901]: pgmap v492: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 MiB/s wr, 19 op/s 2024-03-16T12:24:26.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:25 smithi173 ceph-mon[36632]: pgmap v492: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 21 MiB/s wr, 19 op/s 2024-03-16T12:24:26.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T12:24:26.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T12:24:27.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 -- 172.21.15.98:0/583347871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 msgr2=0x7fceb8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 --2- 172.21.15.98:0/583347871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb8109c30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009900 tx=0x7fceb002bc40 comp rx=0 tx=0).stop 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 -- 172.21.15.98:0/583347871 shutdown_connections 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 --2- 172.21.15.98:0/583347871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb8109c30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 --2- 172.21.15.98:0/583347871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 --2- 172.21.15.98:0/583347871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 0x7fceb8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 -- 172.21.15.98:0/583347871 >> 172.21.15.98:0/583347871 conn(0x7fceb80fcd60 msgr2=0x7fceb80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 -- 172.21.15.98:0/583347871 shutdown_connections 2024-03-16T12:24:27.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.156+0000 7fcebd000640 1 -- 172.21.15.98:0/583347871 wait complete. 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 Processor -- start 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 -- start start 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 0x7fceb8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fceb810d450 con 0x7fceb81017f0 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fceb810d2d0 con 0x7fceb81029e0 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.157+0000 7fcebd000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fceb810d5d0 con 0x7fceb81033a0 2024-03-16T12:24:27.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 0x7fceb8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58400/0 (socket says 172.21.15.98:58400) 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 -- 172.21.15.98:0/3617783771 learned_addr learned my addr 172.21.15.98:0/3617783771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fceb6800640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 -- 172.21.15.98:0/3617783771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 msgr2=0x7fceb819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 -- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 msgr2=0x7fceb8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 0x7fceb8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 -- 172.21.15.98:0/3617783771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb819f6e0 con 0x7fceb81029e0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fceb6800640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.158+0000 7fcead400640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9f0 tx=0x7fcea000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.159+0000 7fceaf400640 1 -- 172.21.15.98:0/3617783771 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea000cd80 con 0x7fceb81029e0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.159+0000 7fceaf400640 1 -- 172.21.15.98:0/3617783771 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea0010910 con 0x7fceb81029e0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.159+0000 7fceaf400640 1 -- 172.21.15.98:0/3617783771 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0018990 con 0x7fceb81029e0 2024-03-16T12:24:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.159+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb819f9d0 con 0x7fceb81029e0 2024-03-16T12:24:27.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.159+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb819ff70 con 0x7fceb81029e0 2024-03-16T12:24:27.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.162+0000 7fceaf400640 1 -- 172.21.15.98:0/3617783771 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcea0010ac0 con 0x7fceb81029e0 2024-03-16T12:24:27.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.162+0000 7fcebfe33740 1 Processor -- start 2024-03-16T12:24:27.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.162+0000 7fcebfe33740 1 -- start start 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff7200 0x56084effbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 0x56084effe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56084ef5b1f0 con 0x56084eff7200 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56084ef03a80 con 0x56084effc370 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcebfe33740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56084ef57290 con 0x56084eff6810 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34392/0 (socket says 172.21.15.98:34392) 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 -- 172.21.15.98:0/821839686 learned_addr learned my addr 172.21.15.98:0/821839686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb5e00640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff7200 0x56084effbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcead400640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 0x56084effe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 -- 172.21.15.98:0/821839686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 msgr2=0x56084effe800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 0x56084effe800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 -- 172.21.15.98:0/821839686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff7200 msgr2=0x56084effbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff7200 0x56084effbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fceb6800640 1 -- 172.21.15.98:0/821839686 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56084ed397d0 con 0x56084eff6810 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.163+0000 7fcead400640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 0x56084effe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.164+0000 7fceaf400640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 0x7fce9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.164+0000 7fceb6800640 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009230 tx=0x7fceb002cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f1848826a53ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.164+0000 7fceaf400640 1 -- 172.21.15.98:0/3617783771 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcea00cf0d0 con 0x7fceb81029e0 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.164+0000 7fceb5e00640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 0x7fce9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.164+0000 7fcea7400640 1 -- 172.21.15.98:0/821839686 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0042070 con 0x56084eff6810 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcea7400640 1 -- 172.21.15.98:0/821839686 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb0004ba0 con 0x56084eff6810 2024-03-16T12:24:27.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcea7400640 1 -- 172.21.15.98:0/821839686 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0005310 con 0x56084eff6810 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 -- 172.21.15.98:0/821839686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 msgr2=0x56084eff6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fceb5e00640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 0x7fce9c0a4d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56084eff7600 tx=0x7fcea8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009230 tx=0x7fceb002cd40 comp rx=0 tx=0).stop 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 -- 172.21.15.98:0/821839686 shutdown_connections 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084effc370 0x56084effe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff7200 0x56084effbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 --2- 172.21.15.98:0/821839686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff6810 0x56084eff6c30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 -- 172.21.15.98:0/821839686 >> 172.21.15.98:0/821839686 conn(0x56084eff1d80 msgr2=0x56084eff2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 -- 172.21.15.98:0/821839686 shutdown_connections 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.165+0000 7fcebfe33740 1 -- 172.21.15.98:0/821839686 wait complete. 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.166+0000 7fcebfe33740 1 -- start start 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.166+0000 7fcebfe33740 1 Processor -- start 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f000d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f008280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56084efff2c0 con 0x56084eff6810 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56084ef16ed0 con 0x56084eff7200 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56084efff440 con 0x56084effc370 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcebfe33740 10 client.0 _ll_register_callbacks cb 0x56084ef52220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f000d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fceb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f008280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58416/0 (socket says 172.21.15.98:58416) 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 -- 172.21.15.98:0/1389772565 learned_addr learned my addr 172.21.15.98:0/1389772565 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 -- 172.21.15.98:0/1389772565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 msgr2=0x56084f008280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f008280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 -- 172.21.15.98:0/1389772565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 msgr2=0x56084f000d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f000d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 -- 172.21.15.98:0/1389772565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56084ec6f3f0 con 0x56084eff7200 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fceb6800640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f008280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fcead400640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcea00cc810 tx=0x7fcea009b2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b4b1f6594c9059b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.167+0000 7fce9be00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea009b760 con 0x56084eff7200 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcea00d1070 con 0x56084eff7200 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb0004ba0 con 0x56084eff7200 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcea009f800 con 0x56084eff7200 2024-03-16T12:24:27.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 10 client.26024 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56084ec2f0f0 con 0x56084eff7200 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb0004ba0 con 0x56084eff7200 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcea009abb0 con 0x56084eff7200 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 10 client.26024 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 10 client.26024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 20 client.26024 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f00cb10 0x56084f00ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 4 client.26024 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.168+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56084ef6ebe0 con 0x56084f00cb10 2024-03-16T12:24:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.169+0000 7fceb5e00640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f00cb10 0x56084f00ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.169+0000 7fceb5e00640 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f00cb10 0x56084f00ef90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c776df357cbceed server_cookie=802d5fb6451a1ba1 in_seq=0 out_seq=0 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.169+0000 7fce9be00640 10 client.26024 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fce9be00640 1 -- 172.21.15.98:0/1389772565 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56084ef6ebe0 con 0x56084f00cb10 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fce9be00640 10 client.26024 handle_command_reply: tid=0 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 client.26024 shutdown 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 msgr2=0x56084f001280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcea00cc810 tx=0x7fcea009b2a0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 shutdown_connections 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f00cb10 0x56084f00ef90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f008280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f001280 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fce9be00640 0 client.26024 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1389772565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f000d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 >> 172.21.15.98:0/1389772565 conn(0x56084eff1d80 msgr2=0x56084eff0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 shutdown_connections 2024-03-16T12:24:27.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.170+0000 7fcebfe33740 1 -- 172.21.15.98:0/1389772565 wait complete. 2024-03-16T12:24:27.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.171+0000 7fcebfe33740 20 client.26024 trim_cache size 0 max 16384 2024-03-16T12:24:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 Processor -- start 2024-03-16T12:24:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 -- start start 2024-03-16T12:24:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff7200 0x56084f008150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084effc370 0x56084eff0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81acc30 con 0x56084effc370 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81aa500 con 0x56084eff6810 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.189+0000 7fcebfe33740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81a8350 con 0x56084eff7200 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58424/0 (socket says 172.21.15.98:58424) 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 -- 172.21.15.98:0/522766497 learned_addr learned my addr 172.21.15.98:0/522766497 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fcead400640 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff7200 0x56084f008150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb6800640 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084effc370 0x56084eff0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 -- 172.21.15.98:0/522766497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff7200 msgr2=0x56084f008150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff7200 0x56084f008150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 -- 172.21.15.98:0/522766497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084effc370 msgr2=0x56084eff0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084effc370 0x56084eff0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 -- 172.21.15.98:0/522766497 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56084eff8de0 con 0x56084eff6810 2024-03-16T12:24:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fceb5e00640 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcea8005fd0 tx=0x7fcea801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6afec00c49eb5d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.190+0000 7fcea6a00640 1 -- 172.21.15.98:0/522766497 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8004550 con 0x56084eff6810 2024-03-16T12:24:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcea6a00640 1 -- 172.21.15.98:0/522766497 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea80046f0 con 0x56084eff6810 2024-03-16T12:24:27.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcea6a00640 1 -- 172.21.15.98:0/522766497 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea800f920 con 0x56084eff6810 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 -- 172.21.15.98:0/522766497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 msgr2=0x56084eff2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcea8005fd0 tx=0x7fcea801a000 comp rx=0 tx=0).stop 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 -- 172.21.15.98:0/522766497 shutdown_connections 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084effc370 0x56084eff0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084eff7200 0x56084f008150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 --2- 172.21.15.98:0/522766497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff6810 0x56084eff2160 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 -- 172.21.15.98:0/522766497 >> 172.21.15.98:0/522766497 conn(0x56084eff1d80 msgr2=0x56084eff0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 -- 172.21.15.98:0/522766497 shutdown_connections 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.191+0000 7fcebfe33740 1 -- 172.21.15.98:0/522766497 wait complete. 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 -- start start 2024-03-16T12:24:27.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 Processor -- start 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f007400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81a89c0 con 0x56084eff6810 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81a9e30 con 0x56084eff7200 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81a9090 con 0x56084effc370 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.192+0000 7fcebfe33740 10 client.0 _ll_register_callbacks cb 0x7fcea0097ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fceb5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fceb6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f007400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58432/0 (socket says 172.21.15.98:58432) 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 -- 172.21.15.98:0/1236058694 learned_addr learned my addr 172.21.15.98:0/1236058694 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 -- 172.21.15.98:0/1236058694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 msgr2=0x56084f007400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f007400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 -- 172.21.15.98:0/1236058694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 msgr2=0x56084f005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 -- 172.21.15.98:0/1236058694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56084efffde0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcead400640 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9c0 tx=0x7fcea0010430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58d3da3215e60a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fce9be00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea009b0c0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcea80046f0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcea009abb0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcea009adb0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.193+0000 7fcebfe33740 10 client.26030 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56084f0733f0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea80046f0 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcea0097540 con 0x56084eff7200 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 10 client.26030 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 10 client.26030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 20 client.26030 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f073d20 0x56084f076140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 4 client.26030 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56084ef68730 con 0x56084f073d20 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.194+0000 7fceb5e00640 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f073d20 0x56084f076140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fceb5e00640 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f073d20 0x56084f076140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21e2e04ae4902d1 server_cookie=813b3bfa2ad2945e in_seq=0 out_seq=0 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fce9be00640 10 client.26030 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fce9be00640 1 -- 172.21.15.98:0/1236058694 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56084ef68730 con 0x56084f073d20 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fce9be00640 10 client.26030 handle_command_reply: tid=0 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 client.26030 shutdown 2024-03-16T12:24:27.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 msgr2=0x56084f0063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9c0 tx=0x7fcea0010430 comp rx=0 tx=0).stop 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 shutdown_connections 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x56084f073d20 0x56084f076140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56084effc370 0x56084f007400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56084eff7200 0x56084f0063f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 --2- 172.21.15.98:0/1236058694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56084eff6810 0x56084f005eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 >> 172.21.15.98:0/1236058694 conn(0x56084eff1d80 msgr2=0x56084eff0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fce9be00640 0 client.26030 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 shutdown_connections 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 1 -- 172.21.15.98:0/1236058694 wait complete. 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.195+0000 7fcebfe33740 20 client.26030 trim_cache size 0 max 16384 2024-03-16T12:24:27.179 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-03-16T12:24:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-03-16T12:24:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:27.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 msgr2=0x7fce9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 0x7fce9c0a4d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56084eff7600 tx=0x7fcea8009500 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 msgr2=0x7fceb8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcea000e9f0 tx=0x7fcea000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 shutdown_connections 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fce9c0a2880 0x7fce9c0a4d40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fceb81033a0 0x7fceb819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fceb81029e0 0x7fceb8198000 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 --2- 172.21.15.98:0/3617783771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fceb81017f0 0x7fceb8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 >> 172.21.15.98:0/3617783771 conn(0x7fceb80fcd60 msgr2=0x7fceb8105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 shutdown_connections 2024-03-16T12:24:27.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.196+0000 7fcebd000640 1 -- 172.21.15.98:0/3617783771 wait complete. 2024-03-16T12:24:27.192 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T12:24:27.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 -- 172.21.15.98:0/948221924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 msgr2=0x7f2fb810b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 --2- 172.21.15.98:0/948221924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb810b910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009ca0 tx=0x7f2fb002c190 comp rx=0 tx=0).stop 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 -- 172.21.15.98:0/948221924 shutdown_connections 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 --2- 172.21.15.98:0/948221924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb810b910 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 --2- 172.21.15.98:0/948221924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80741e0 0x7f2fb8074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 --2- 172.21.15.98:0/948221924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80757c0 0x7f2fb8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 -- 172.21.15.98:0/948221924 >> 172.21.15.98:0/948221924 conn(0x7f2fb80fcb70 msgr2=0x7f2fb80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 -- 172.21.15.98:0/948221924 shutdown_connections 2024-03-16T12:24:27.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.423+0000 7f2fb7200640 1 -- 172.21.15.98:0/948221924 wait complete. 2024-03-16T12:24:27.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.424+0000 7f2fb7200640 1 Processor -- start 2024-03-16T12:24:27.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 -- start start 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80757c0 0x7f2fb8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb81a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb810f440 con 0x7f2fb80757c0 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb810f2c0 con 0x7f2fb80741e0 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb810f5c0 con 0x7f2fb8109470 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58438/0 (socket says 172.21.15.98:58438) 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb5e00640 1 -- 172.21.15.98:0/2379898818 learned_addr learned my addr 172.21.15.98:0/2379898818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fad400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80757c0 0x7f2fb8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb6800640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb81a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb5e00640 1 -- 172.21.15.98:0/2379898818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 msgr2=0x7f2fb81a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.425+0000 7f2fb5e00640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb81a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb5e00640 1 -- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80757c0 msgr2=0x7f2fb8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb5e00640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80757c0 0x7f2fb8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb5e00640 1 -- 172.21.15.98:0/2379898818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb81a1670 con 0x7f2fb80741e0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb5e00640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e9f0 tx=0x7f2fa800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800cd80 con 0x7f2fb80741e0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fa8010910 con 0x7f2fb80741e0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8018990 con 0x7f2fb80741e0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb81a1960 con 0x7f2fb80741e0 2024-03-16T12:24:27.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.426+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb81a1e60 con 0x7f2fb80741e0 2024-03-16T12:24:27.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.427+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb8068940 con 0x7f2fb80741e0 2024-03-16T12:24:27.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.429+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2fa8010ac0 con 0x7f2fb80741e0 2024-03-16T12:24:27.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.430+0000 7f2faf400640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 0x7f2f9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.430+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2fa80cf0d0 con 0x7f2fb80741e0 2024-03-16T12:24:27.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.431+0000 7f2fad400640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 0x7f2f9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.431+0000 7f2fad400640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 0x7f2f9c0a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2fb819af60 tx=0x7f2fa0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.434+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2fa809b760 con 0x7f2fb80741e0 2024-03-16T12:24:27.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.671+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fb8108280 con 0x7f2fb80741e0 2024-03-16T12:24:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.672+0000 7f2faf400640 1 -- 172.21.15.98:0/2379898818 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7f2fb8108280 con 0x7f2fb80741e0 2024-03-16T12:24:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:24:27.652 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:24:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 msgr2=0x7f2f9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 0x7f2f9c0a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2fb819af60 tx=0x7f2fa0005f90 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 msgr2=0x7f2fb8199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e9f0 tx=0x7f2fa800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 shutdown_connections 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f2f9c0a2880 0x7f2f9c0a4d40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fb8109470 0x7f2fb81a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fb80757c0 0x7f2fb8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.676+0000 7f2fb7200640 1 --2- 172.21.15.98:0/2379898818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fb80741e0 0x7f2fb8199a50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.677+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 >> 172.21.15.98:0/2379898818 conn(0x7f2fb80fcb70 msgr2=0x7f2fb8105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.677+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 shutdown_connections 2024-03-16T12:24:27.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.677+0000 7f2fb7200640 1 -- 172.21.15.98:0/2379898818 wait complete. 2024-03-16T12:24:27.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 -- 172.21.15.98:0/818217306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e8101800 msgr2=0x7fb1e8101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 --2- 172.21.15.98:0/818217306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e8101800 0x7fb1e8101c20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009900 tx=0x7fb1d802bc30 comp rx=0 tx=0).stop 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 -- 172.21.15.98:0/818217306 shutdown_connections 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 --2- 172.21.15.98:0/818217306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e81033b0 0x7fb1e8109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 --2- 172.21.15.98:0/818217306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81029f0 0x7fb1e8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 --2- 172.21.15.98:0/818217306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e8101800 0x7fb1e8101c20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 -- 172.21.15.98:0/818217306 >> 172.21.15.98:0/818217306 conn(0x7fb1e80fcd70 msgr2=0x7fb1e80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 -- 172.21.15.98:0/818217306 shutdown_connections 2024-03-16T12:24:27.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.908+0000 7fb1edc00640 1 -- 172.21.15.98:0/818217306 wait complete. 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 Processor -- start 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 -- start start 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 0x7fb1e8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 0x7fb1e819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 0x7fb1e81a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810d4c0 con 0x7fb1e8101800 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810d340 con 0x7fb1e81033b0 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1edc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810d640 con 0x7fb1e81029f0 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1e6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 0x7fb1e819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.909+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 0x7fb1e8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 0x7fb1e81a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 0x7fb1e819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34442/0 (socket says 172.21.15.98:34442) 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6000640 1 -- 172.21.15.98:0/3943735266 learned_addr learned my addr 172.21.15.98:0/3943735266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 -- 172.21.15.98:0/3943735266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 msgr2=0x7fb1e819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 0x7fb1e819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 -- 172.21.15.98:0/3943735266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 msgr2=0x7fb1e81a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 0x7fb1e81a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 -- 172.21.15.98:0/3943735266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e81a1950 con 0x7fb1e8101800 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e7400640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 0x7fb1e81a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:27.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.910+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 0x7fb1e8199d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009230 tx=0x7fb1d802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.911+0000 7fb1df400640 1 -- 172.21.15.98:0/3943735266 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8043020 con 0x7fb1e8101800 2024-03-16T12:24:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.911+0000 7fb1df400640 1 -- 172.21.15.98:0/3943735266 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d80054e0 con 0x7fb1e8101800 2024-03-16T12:24:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.911+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e81a1be0 con 0x7fb1e8101800 2024-03-16T12:24:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.911+0000 7fb1df400640 1 -- 172.21.15.98:0/3943735266 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8003b60 con 0x7fb1e8101800 2024-03-16T12:24:27.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.911+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e81a2100 con 0x7fb1e8101800 2024-03-16T12:24:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.915+0000 7fb1f0a23740 1 Processor -- start 2024-03-16T12:24:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.915+0000 7fb1f0a23740 1 -- start start 2024-03-16T12:24:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.915+0000 7fb1df400640 1 -- 172.21.15.98:0/3943735266 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb1d8046020 con 0x7fb1e8101800 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f4fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e15f173b0 con 0x560e15f4f8c0 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e15eb4450 con 0x560e15f502b0 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e15e5ccb0 con 0x560e15f55420 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f4fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58462/0 (socket says 172.21.15.98:58462) 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2082317646 learned_addr learned my addr 172.21.15.98:0/2082317646 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2082317646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 msgr2=0x560e15f578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2082317646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 msgr2=0x560e15f4fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f4fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.916+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2082317646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e15e7e1f0 con 0x560e15f502b0 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80f4d90 tx=0x7fb1d80f4dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c1ed92f2a24201 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1df400640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1d7e00640 1 -- 172.21.15.98:0/2082317646 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80f7b10 con 0x560e15f502b0 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1d7e00640 1 -- 172.21.15.98:0/2082317646 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d80bdb50 con 0x560e15f502b0 2024-03-16T12:24:27.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1d7e00640 1 -- 172.21.15.98:0/2082317646 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80bcdb0 con 0x560e15f502b0 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1df400640 1 -- 172.21.15.98:0/3943735266 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1d80f0e70 con 0x7fb1e8101800 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.917+0000 7fb1e7400640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2082317646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 msgr2=0x560e15f54db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80f4d90 tx=0x7fb1d80f4dc0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1e7400640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x560e15f50b10 tx=0x7fb1e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2082317646 shutdown_connections 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f54db0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2082317646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f4fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2082317646 >> 172.21.15.98:0/2082317646 conn(0x560e15f4ae30 msgr2=0x560e15f4b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2082317646 shutdown_connections 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2082317646 wait complete. 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.918+0000 7fb1f0a23740 1 -- start start 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 Processor -- start 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f59840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 0x560e15f59d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e15e70110 con 0x560e15f4f8c0 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e15eb04f0 con 0x560e15f55420 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e15f58190 con 0x560e15f502b0 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1f0a23740 10 client.0 _ll_register_callbacks cb 0x560e15e5db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f59840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58470/0 (socket says 172.21.15.98:58470) 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1e7400640 1 -- 172.21.15.98:0/1184159370 learned_addr learned my addr 172.21.15.98:0/1184159370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.919+0000 7fb1e6000640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 0x560e15f59d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 -- 172.21.15.98:0/1184159370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 msgr2=0x560e15f59d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 0x560e15f59d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 -- 172.21.15.98:0/1184159370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 msgr2=0x560e15f59840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f59840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 -- 172.21.15.98:0/1184159370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e15bc83f0 con 0x560e15f55420 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e6000640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 0x560e15f59d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1e7400640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0004650 tx=0x7fb1e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499213b2044f1984 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000f830 con 0x560e15f55420 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d80bdb50 con 0x560e15f55420 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.920+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1e000f9d0 con 0x560e15f55420 2024-03-16T12:24:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb1e000fbd0 con 0x560e15f55420 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 10 client.26048 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560e15c927d0 con 0x560e15f55420 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d80bdb50 con 0x560e15f55420 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb1e0027070 con 0x560e15f55420 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 10 client.26048 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 10 client.26048 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 20 client.26048 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15f65740 0x560e15f67bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 4 client.26048 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e15ec1990 con 0x560e15f65740 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.921+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15f65740 0x560e15f67bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.922+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15f65740 0x560e15f67bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7dc867f769237d4f server_cookie=85e7c7bd5037c887 in_seq=0 out_seq=0 2024-03-16T12:24:27.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.922+0000 7fb1d4c00640 10 client.26048 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:24:27.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1d4c00640 1 -- 172.21.15.98:0/1184159370 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e15ec1990 con 0x560e15f65740 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1d4c00640 10 client.26048 handle_command_reply: tid=0 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 client.26048 shutdown 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 msgr2=0x560e15f60dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0004650 tx=0x7fb1e0004730 comp rx=0 tx=0).stop 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 shutdown_connections 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15f65740 0x560e15f67bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f55420 0x560e15f60dc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f502b0 0x560e15f59d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/1184159370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f4f8c0 0x560e15f59840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 >> 172.21.15.98:0/1184159370 conn(0x560e15f4ae30 msgr2=0x560e15f4a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1d4c00640 0 client.26048 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:24:27.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 shutdown_connections 2024-03-16T12:24:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 1 -- 172.21.15.98:0/1184159370 wait complete. 2024-03-16T12:24:27.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.923+0000 7fb1f0a23740 20 client.26048 trim_cache size 0 max 16384 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 Processor -- start 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 -- start start 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 0x560e15f51670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f55420 0x560e15f60d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81aed70 con 0x560e15f55420 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81ac640 con 0x560e15f502b0 2024-03-16T12:24:27.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81aa490 con 0x560e15f4f8c0 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1e6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f55420 0x560e15f60d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1e6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58478/0 (socket says 172.21.15.98:58478) 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 0x560e15f51670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.938+0000 7fb1e6000640 1 -- 172.21.15.98:0/2156821864 learned_addr learned my addr 172.21.15.98:0/2156821864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2156821864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 msgr2=0x560e15f51bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2156821864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f55420 msgr2=0x560e15f60d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f55420 0x560e15f60d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 -- 172.21.15.98:0/2156821864 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e15f46c10 con 0x560e15f4f8c0 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6000640 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 0x560e15f51670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x560e15f506b0 tx=0x7fb1d8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce568906ed25952a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1d7400640 1 -- 172.21.15.98:0/2156821864 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80d1830 con 0x560e15f4f8c0 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1d7400640 1 -- 172.21.15.98:0/2156821864 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d80f4680 con 0x560e15f4f8c0 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.939+0000 7fb1d7400640 1 -- 172.21.15.98:0/2156821864 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80d1a00 con 0x560e15f4f8c0 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2156821864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 msgr2=0x560e15f51670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 0x560e15f51670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x560e15f506b0 tx=0x7fb1d8000bc0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2156821864 shutdown_connections 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f55420 0x560e15f60d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f502b0 0x560e15f51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/2156821864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f4f8c0 0x560e15f51670 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2156821864 >> 172.21.15.98:0/2156821864 conn(0x560e15f4ae30 msgr2=0x560e15f4b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2156821864 shutdown_connections 2024-03-16T12:24:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- 172.21.15.98:0/2156821864 wait complete. 2024-03-16T12:24:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 -- start start 2024-03-16T12:24:27.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.940+0000 7fb1f0a23740 1 Processor -- start 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f502b0 0x560e15f5e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f5f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81aab00 con 0x560e15f502b0 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81abf70 con 0x560e15f4f8c0 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e81ab1d0 con 0x560e15f55420 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1f0a23740 10 client.0 _ll_register_callbacks cb 0x7fb1e0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58484/0 (socket says 172.21.15.98:58484) 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 -- 172.21.15.98:0/4030487365 learned_addr learned my addr 172.21.15.98:0/4030487365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:27.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e7400640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f5f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6000640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f502b0 0x560e15f5e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 -- 172.21.15.98:0/4030487365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 msgr2=0x560e15f5f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f5f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 -- 172.21.15.98:0/4030487365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f502b0 msgr2=0x560e15f5e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f502b0 0x560e15f5e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.941+0000 7fb1e6a00640 1 -- 172.21.15.98:0/4030487365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e15f4bba0 con 0x560e15f4f8c0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1e6a00640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80d2100 tx=0x7fb1d8005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707cc0e65ac771c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80d1be0 con 0x560e15f4f8c0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1d80bb750 con 0x560e15f4f8c0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d80f4680 con 0x560e15f4f8c0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb1d80d1f60 con 0x560e15f4f8c0 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 10 client.26054 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:27.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560e15f60be0 con 0x560e15f4f8c0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d80f4680 con 0x560e15f4f8c0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb1d80b9a30 con 0x560e15f4f8c0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 10 client.26054 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 10 client.26054 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 20 client.26054 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15fcbaf0 0x560e15fcdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 4 client.26054 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.942+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 --> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e15ebbce0 con 0x560e15fcbaf0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1e6000640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15fcbaf0 0x560e15fcdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1e6000640 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15fcbaf0 0x560e15fcdee0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d3e4360907be5c1 server_cookie=db772428e7f4e075 in_seq=0 out_seq=0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1d4c00640 10 client.26054 ms_handle_connect on v2:172.21.15.153:6836/419588181 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1d4c00640 1 -- 172.21.15.98:0/4030487365 <== mds.2 v2:172.21.15.153:6836/419588181 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e15ebbce0 con 0x560e15fcbaf0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1d4c00640 10 client.26054 handle_command_reply: tid=0 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.943+0000 7fb1f0a23740 1 client.26054 shutdown 2024-03-16T12:24:27.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 msgr2=0x560e15f5df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80d2100 tx=0x7fb1d8005e10 comp rx=0 tx=0).stop 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 shutdown_connections 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:6836/419588181,v1:172.21.15.153:6837/419588181] conn(0x560e15fcbaf0 0x560e15fcdee0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e15f55420 0x560e15f5f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e15f502b0 0x560e15f5e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 --2- 172.21.15.98:0/4030487365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e15f4f8c0 0x560e15f5df70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1d4c00640 0 client.26054 ms_handle_reset on v2:172.21.15.153:6836/419588181 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 >> 172.21.15.98:0/4030487365 conn(0x560e15f4ae30 msgr2=0x560e15f59800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 shutdown_connections 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 1 -- 172.21.15.98:0/4030487365 wait complete. 2024-03-16T12:24:27.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1f0a23740 20 client.26054 trim_cache size 0 max 16384 2024-03-16T12:24:27.929 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 msgr2=0x7fb1c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.944+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x560e15f50b10 tx=0x7fb1e0005f90 comp rx=0 tx=0).stop 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 msgr2=0x7fb1e8199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 0x7fb1e8199d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009230 tx=0x7fb1d802c5d0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 shutdown_connections 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81033b0 0x7fb1e81a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1e81029f0 0x7fb1e819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 --2- 172.21.15.98:0/3943735266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e8101800 0x7fb1e8199d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 >> 172.21.15.98:0/3943735266 conn(0x7fb1e80fcd70 msgr2=0x7fb1e8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 shutdown_connections 2024-03-16T12:24:27.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:27.945+0000 7fb1edc00640 1 -- 172.21.15.98:0/3943735266 wait complete. 2024-03-16T12:24:27.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T12:24:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 -- 172.21.15.98:0/2707319608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf28102290 msgr2=0x7fcf28108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 --2- 172.21.15.98:0/2707319608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf28102290 0x7fcf28108b20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009ca0 tx=0x7fcf2002c100 comp rx=0 tx=0).stop 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 -- 172.21.15.98:0/2707319608 shutdown_connections 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 --2- 172.21.15.98:0/2707319608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf28102290 0x7fcf28108b20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 --2- 172.21.15.98:0/2707319608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 0x7fcf28101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 --2- 172.21.15.98:0/2707319608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281006e0 0x7fcf28100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 -- 172.21.15.98:0/2707319608 >> 172.21.15.98:0/2707319608 conn(0x7fcf280fbc70 msgr2=0x7fcf280fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 -- 172.21.15.98:0/2707319608 shutdown_connections 2024-03-16T12:24:28.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.167+0000 7fcf2d200640 1 -- 172.21.15.98:0/2707319608 wait complete. 2024-03-16T12:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 Processor -- start 2024-03-16T12:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 -- start start 2024-03-16T12:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 0x7fcf28102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf28102290 0x7fcf281031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810c340 con 0x7fcf281018d0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810c1c0 con 0x7fcf281006e0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.168+0000 7fcf2d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810c4c0 con 0x7fcf28102290 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58508/0 (socket says 172.21.15.98:58508) 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 -- 172.21.15.98:0/855844834 learned_addr learned my addr 172.21.15.98:0/855844834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25400640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 0x7fcf28102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf26800640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf28102290 0x7fcf281031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 -- 172.21.15.98:0/855844834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf28102290 msgr2=0x7fcf281031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf28102290 0x7fcf281031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 -- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 msgr2=0x7fcf28102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 0x7fcf28102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 -- 172.21.15.98:0/855844834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf28103a80 con 0x7fcf281006e0 2024-03-16T12:24:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.169+0000 7fcf25e00640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800e9c0 tx=0x7fcf1800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.170+0000 7fcf1ea00640 1 -- 172.21.15.98:0/855844834 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800cd80 con 0x7fcf281006e0 2024-03-16T12:24:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.170+0000 7fcf1ea00640 1 -- 172.21.15.98:0/855844834 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf180035f0 con 0x7fcf281006e0 2024-03-16T12:24:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.170+0000 7fcf1ea00640 1 -- 172.21.15.98:0/855844834 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18010690 con 0x7fcf281006e0 2024-03-16T12:24:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.170+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf281a1b70 con 0x7fcf281006e0 2024-03-16T12:24:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.170+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf281a2110 con 0x7fcf281006e0 2024-03-16T12:24:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf1ea00640 1 -- 172.21.15.98:0/855844834 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcf180037a0 con 0x7fcf281006e0 2024-03-16T12:24:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 Processor -- start 2024-03-16T12:24:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 -- start start 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e48ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e4ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b9e103b0 con 0x5633b9e488c0 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5633b9dad450 con 0x5633b9e492b0 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.173+0000 7fcf30010740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633b9d55cb0 con 0x5633b9e4e420 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf26800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e48ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e4ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf26800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e48ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55896/0 (socket says 172.21.15.98:55896) 2024-03-16T12:24:28.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34496/0 (socket says 172.21.15.98:34496) 2024-03-16T12:24:28.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf26800640 1 -- 172.21.15.98:0/3768660095 learned_addr learned my addr 172.21.15.98:0/3768660095 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 -- 172.21.15.98:0/3768660095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 msgr2=0x5633b9e508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 -- 172.21.15.98:0/3768660095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 msgr2=0x5633b9e48ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e48ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.174+0000 7fcf25e00640 1 -- 172.21.15.98:0/3768660095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b9d771f0 con 0x5633b9e492b0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf25e00640 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e4ddb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcf180cc890 tx=0x7fcf180d7640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7064f8f928991e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf1ea00640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 0x7fcf0c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf13400640 1 -- 172.21.15.98:0/3768660095 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf180e2070 con 0x5633b9e492b0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf13400640 1 -- 172.21.15.98:0/3768660095 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf180d7e40 con 0x5633b9e492b0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf13400640 1 -- 172.21.15.98:0/3768660095 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18004000 con 0x5633b9e492b0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf26800640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 0x7fcf0c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.175+0000 7fcf1ea00640 1 -- 172.21.15.98:0/855844834 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcf180cf1b0 con 0x7fcf281006e0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 -- 172.21.15.98:0/3768660095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 msgr2=0x5633b9e4ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e4ddb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcf180cc890 tx=0x7fcf180d7640 comp rx=0 tx=0).stop 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf26800640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 0x7fcf0c0a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcf20006010 tx=0x7fcf20032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 -- 172.21.15.98:0/3768660095 shutdown_connections 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e4ddb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 --2- 172.21.15.98:0/3768660095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e48ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 -- 172.21.15.98:0/3768660095 >> 172.21.15.98:0/3768660095 conn(0x5633b9e43e30 msgr2=0x5633b9e44230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 -- 172.21.15.98:0/3768660095 shutdown_connections 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.176+0000 7fcf30010740 1 -- 172.21.15.98:0/3768660095 wait complete. 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 -- start start 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 Processor -- start 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e53100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e5a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5633b9d69110 con 0x5633b9e492b0 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5633b9da94f0 con 0x5633b9e488c0 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633b9e513d0 con 0x5633b9e4e420 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf30010740 10 client.0 _ll_register_callbacks cb 0x5633b9d56b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:28.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58520/0 (socket says 172.21.15.98:58520) 2024-03-16T12:24:28.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf25400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e53100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf25e00640 1 -- 172.21.15.98:0/4209779836 learned_addr learned my addr 172.21.15.98:0/4209779836 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.177+0000 7fcf26800640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e5a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 -- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 msgr2=0x5633b9e5a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e5a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 -- 172.21.15.98:0/4209779836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 msgr2=0x5633b9e53100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e53100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 -- 172.21.15.98:0/4209779836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b9ac13f0 con 0x5633b9e488c0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf25e00640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcf1809b720 tx=0x7fcf180d2dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=933655b95b9516ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf07e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18017b10 con 0x5633b9e488c0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcf180cca00 con 0x5633b9e488c0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.178+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633b9dba990 con 0x5633b9e488c0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcf1800f740 con 0x5633b9e488c0 2024-03-16T12:24:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 10 client.26066 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633b9b8b7d0 con 0x5633b9e488c0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5633b9dba990 con 0x5633b9e488c0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcf18097e00 con 0x5633b9e488c0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 10 client.26066 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 10 client.26066 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 20 client.26066 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9e5e630 0x5633b9e60a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 4 client.26066 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633b9db4ce0 con 0x5633b9e5e630 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.179+0000 7fcf25400640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9e5e630 0x5633b9e60a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.180+0000 7fcf25400640 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9e5e630 0x5633b9e60a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=def5d93e413f3cc2 server_cookie=6081568614aafe6 in_seq=0 out_seq=0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.180+0000 7fcf07e00640 10 client.26066 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf07e00640 1 -- 172.21.15.98:0/4209779836 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633b9db4ce0 con 0x5633b9e5e630 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf07e00640 10 client.26066 handle_command_reply: tid=0 2024-03-16T12:24:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 client.26066 shutdown 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 msgr2=0x5633b9e52bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcf1809b720 tx=0x7fcf180d2dc0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 shutdown_connections 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9e5e630 0x5633b9e60a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e5a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e53100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 --2- 172.21.15.98:0/4209779836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e52bc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf07e00640 0 client.26066 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 >> 172.21.15.98:0/4209779836 conn(0x5633b9e43e30 msgr2=0x5633b9e431f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 shutdown_connections 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 1 -- 172.21.15.98:0/4209779836 wait complete. 2024-03-16T12:24:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.181+0000 7fcf30010740 20 client.26066 trim_cache size 0 max 16384 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.195+0000 7fcf30010740 1 Processor -- start 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.195+0000 7fcf30010740 1 -- start start 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281aee00 con 0x5633b9e488c0 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281ac6d0 con 0x5633b9e492b0 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf30010740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281aa520 con 0x5633b9e4e420 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55916/0 (socket says 172.21.15.98:55916) 2024-03-16T12:24:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58536/0 (socket says 172.21.15.98:58536) 2024-03-16T12:24:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25e00640 1 -- 172.21.15.98:0/2594387577 learned_addr learned my addr 172.21.15.98:0/2594387577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf26800640 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 -- 172.21.15.98:0/2594387577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 msgr2=0x5633b9e523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 -- 172.21.15.98:0/2594387577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 msgr2=0x5633b9e44210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25400640 1 -- 172.21.15.98:0/2594387577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b9e3fc10 con 0x5633b9e492b0 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.196+0000 7fcf25e00640 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf25400640 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5633b9e610d0 tx=0x7fcf14005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57befd8e1932dab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf12a00640 1 -- 172.21.15.98:0/2594387577 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401f050 con 0x5633b9e492b0 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf12a00640 1 -- 172.21.15.98:0/2594387577 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf1401f6d0 con 0x5633b9e492b0 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf12a00640 1 -- 172.21.15.98:0/2594387577 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14010c50 con 0x5633b9e492b0 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf30010740 1 -- 172.21.15.98:0/2594387577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 msgr2=0x5633b9e51e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.197+0000 7fcf30010740 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5633b9e610d0 tx=0x7fcf14005f90 comp rx=0 tx=0).stop 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 -- 172.21.15.98:0/2594387577 shutdown_connections 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e492b0 0x5633b9e51e90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 --2- 172.21.15.98:0/2594387577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e488c0 0x5633b9e44210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 -- 172.21.15.98:0/2594387577 >> 172.21.15.98:0/2594387577 conn(0x5633b9e43e30 msgr2=0x5633b9e431f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 -- 172.21.15.98:0/2594387577 shutdown_connections 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 -- 172.21.15.98:0/2594387577 wait complete. 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 -- start start 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 Processor -- start 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.198+0000 7fcf30010740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281aab90 con 0x5633b9e492b0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281ac000 con 0x5633b9e488c0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcf281ab260 con 0x5633b9e4e420 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf30010740 10 client.0 _ll_register_callbacks cb 0x7fcf180e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf26800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58540/0 (socket says 172.21.15.98:58540) 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 -- 172.21.15.98:0/956426508 learned_addr learned my addr 172.21.15.98:0/956426508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 -- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 msgr2=0x5633b9e589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 -- 172.21.15.98:0/956426508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 msgr2=0x5633b9e579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 -- 172.21.15.98:0/956426508 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633b9e58fb0 con 0x5633b9e488c0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf25e00640 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcf1809b490 tx=0x7fcf18004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bb3724d559ade4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.199+0000 7fcf07e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800fa70 con 0x5633b9e488c0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcf18097e00 con 0x5633b9e488c0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf1401f6d0 con 0x5633b9e488c0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcf1800fc50 con 0x5633b9e488c0 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 10 client.26072 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633b9eccfd0 con 0x5633b9e488c0 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf1401f6d0 con 0x5633b9e488c0 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fcf180d2350 con 0x5633b9e488c0 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 10 client.26072 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 10 client.26072 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 20 client.26072 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9ecd6d0 0x5633b9ecfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 4 client.26072 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.200+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 --> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633b9d32560 con 0x5633b9ecd6d0 2024-03-16T12:24:28.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf25400640 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9ecd6d0 0x5633b9ecfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf25400640 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9ecd6d0 0x5633b9ecfac0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6415d94f9c862e63 server_cookie=cd7e93502bfbe866 in_seq=0 out_seq=0 2024-03-16T12:24:28.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf07e00640 10 client.26072 ms_handle_connect on v2:172.21.15.173:6838/205594788 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf07e00640 1 -- 172.21.15.98:0/956426508 <== mds.1 v2:172.21.15.173:6838/205594788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633b9d32560 con 0x5633b9ecd6d0 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf07e00640 10 client.26072 handle_command_reply: tid=0 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 client.26072 shutdown 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 msgr2=0x5633b9e57470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcf1809b490 tx=0x7fcf18004730 comp rx=0 tx=0).stop 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 shutdown_connections 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:6838/205594788,v1:172.21.15.173:6839/205594788] conn(0x5633b9ecd6d0 0x5633b9ecfac0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633b9e4e420 0x5633b9e589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5633b9e492b0 0x5633b9e579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 --2- 172.21.15.98:0/956426508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633b9e488c0 0x5633b9e57470 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf07e00640 0 client.26072 ms_handle_reset on v2:172.21.15.173:6838/205594788 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.201+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 >> 172.21.15.98:0/956426508 conn(0x5633b9e43e30 msgr2=0x5633b9e431f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 shutdown_connections 2024-03-16T12:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf30010740 1 -- 172.21.15.98:0/956426508 wait complete. 2024-03-16T12:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf30010740 20 client.26072 trim_cache size 0 max 16384 2024-03-16T12:24:28.186 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:24:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 msgr2=0x7fcf0c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 0x7fcf0c0a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcf20006010 tx=0x7fcf20032590 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 msgr2=0x7fcf28104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800e9c0 tx=0x7fcf1800ee80 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 shutdown_connections 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fcf0c0a2880 0x7fcf0c0a4d40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf28102290 0x7fcf281031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf281018d0 0x7fcf28102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 --2- 172.21.15.98:0/855844834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf281006e0 0x7fcf28104670 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 >> 172.21.15.98:0/855844834 conn(0x7fcf280fbc70 msgr2=0x7fcf280fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.202+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 shutdown_connections 2024-03-16T12:24:28.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.203+0000 7fcf2d200640 1 -- 172.21.15.98:0/855844834 wait complete. 2024-03-16T12:24:28.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:27 smithi098 ceph-mon[31633]: pgmap v493: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 18 op/s 2024-03-16T12:24:28.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:27 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2379898818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:24:28.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T12:24:28.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:27 smithi153 ceph-mon[36901]: pgmap v493: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 18 op/s 2024-03-16T12:24:28.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:27 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2379898818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:24:28.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:27 smithi173 ceph-mon[36632]: pgmap v493: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 18 op/s 2024-03-16T12:24:28.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:27 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2379898818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:24:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 -- 172.21.15.98:0/697082257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4101350 msgr2=0x7f47b41017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 --2- 172.21.15.98:0/697082257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4101350 0x7f47b41017d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009ca0 tx=0x7f47ac02c100 comp rx=0 tx=0).stop 2024-03-16T12:24:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 -- 172.21.15.98:0/697082257 shutdown_connections 2024-03-16T12:24:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 --2- 172.21.15.98:0/697082257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4101350 0x7f47b41017d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 --2- 172.21.15.98:0/697082257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 --2- 172.21.15.98:0/697082257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4100050 0x7f47b4100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 -- 172.21.15.98:0/697082257 >> 172.21.15.98:0/697082257 conn(0x7f47b40fbc70 msgr2=0x7f47b40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 -- 172.21.15.98:0/697082257 shutdown_connections 2024-03-16T12:24:28.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.408+0000 7f47bb600640 1 -- 172.21.15.98:0/697082257 wait complete. 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 Processor -- start 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 -- start start 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4101350 0x7f47b41a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410f630 con 0x7f47b4100050 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.409+0000 7f47bb600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410f4b0 con 0x7f47b4101350 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47bb600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410f7b0 con 0x7f47b41009b0 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4101350 0x7f47b41a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34540/0 (socket says 172.21.15.98:34540) 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55942/0 (socket says 172.21.15.98:55942) 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 -- 172.21.15.98:0/1280926886 learned_addr learned my addr 172.21.15.98:0/1280926886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 -- 172.21.15.98:0/1280926886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4101350 msgr2=0x7f47b41a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4101350 0x7f47b41a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 -- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 msgr2=0x7f47b4199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 -- 172.21.15.98:0/1280926886 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b41a18b0 con 0x7f47b41009b0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b8c00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.410+0000 7f47b3e00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9f0 tx=0x7f47a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.411+0000 7f47b1600640 1 -- 172.21.15.98:0/1280926886 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a000cd80 con 0x7f47b41009b0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.411+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b41a1ba0 con 0x7f47b41009b0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.411+0000 7f47b1600640 1 -- 172.21.15.98:0/1280926886 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47a0010910 con 0x7f47b41009b0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.411+0000 7f47b1600640 1 -- 172.21.15.98:0/1280926886 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a00188f0 con 0x7f47b41009b0 2024-03-16T12:24:28.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.411+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b41a2140 con 0x7f47b41009b0 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 Processor -- start 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 -- start start 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08632b0 0x55b0a0867db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a086a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0a082a3b0 con 0x55b0a0868420 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b0a07c7450 con 0x55b0a08628c0 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47be262740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0a076fcb0 con 0x55b0a08632b0 2024-03-16T12:24:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.414+0000 7f47b1600640 1 -- 172.21.15.98:0/1280926886 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f47a0010ac0 con 0x7f47b41009b0 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08632b0 0x55b0a0867db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58548/0 (socket says 172.21.15.98:58548) 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 -- 172.21.15.98:0/148999513 learned_addr learned my addr 172.21.15.98:0/148999513 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b3e00640 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a086a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 -- 172.21.15.98:0/148999513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08632b0 msgr2=0x55b0a0867db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08632b0 0x55b0a0867db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 -- 172.21.15.98:0/148999513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 msgr2=0x55b0a086a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a086a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b9600640 1 -- 172.21.15.98:0/148999513 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0a07911f0 con 0x55b0a08628c0 2024-03-16T12:24:28.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.415+0000 7f47b3e00640 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a086a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:24:28.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.416+0000 7f47b9600640 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47ac02c710 tx=0x7f47ac0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=161115289ec42f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.416+0000 7f47a5c00640 1 -- 172.21.15.98:0/148999513 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac043070 con 0x55b0a08628c0 2024-03-16T12:24:28.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.416+0000 7f47a5c00640 1 -- 172.21.15.98:0/148999513 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47ac032c90 con 0x55b0a08628c0 2024-03-16T12:24:28.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.416+0000 7f47a5c00640 1 -- 172.21.15.98:0/148999513 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac039040 con 0x55b0a08628c0 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.416+0000 7f47b1600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 0x7f479c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47b1600640 1 -- 172.21.15.98:0/1280926886 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f47a00cf130 con 0x7f47b41009b0 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47b8c00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 0x7f479c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 -- 172.21.15.98:0/148999513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 msgr2=0x55b0a0862ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47ac02c710 tx=0x7f47ac0323d0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 -- 172.21.15.98:0/148999513 shutdown_connections 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a086a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08632b0 0x55b0a0867db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 --2- 172.21.15.98:0/148999513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0862ce0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 -- 172.21.15.98:0/148999513 >> 172.21.15.98:0/148999513 conn(0x55b0a085de30 msgr2=0x55b0a085e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 -- 172.21.15.98:0/148999513 shutdown_connections 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47be262740 1 -- 172.21.15.98:0/148999513 wait complete. 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.417+0000 7f47b8c00640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 0x7f479c0a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55b0a085eda0 tx=0x7f47a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 -- start start 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 Processor -- start 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a086d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a0868420 0x55b0a0874090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0a0783110 con 0x55b0a08632b0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b0a07c34f0 con 0x55b0a0868420 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0a086b370 con 0x55b0a08628c0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47be262740 10 client.0 _ll_register_callbacks cb 0x55b0a0770b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34552/0 (socket says 172.21.15.98:34552) 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47b8c00640 1 -- 172.21.15.98:0/963776775 learned_addr learned my addr 172.21.15.98:0/963776775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47b9600640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a0868420 0x55b0a0874090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.418+0000 7f47b3e00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a086d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 -- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a0868420 msgr2=0x55b0a0874090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a0868420 0x55b0a0874090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 -- 172.21.15.98:0/963776775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 msgr2=0x55b0a086d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a086d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 -- 172.21.15.98:0/963776775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0a04db3f0 con 0x55b0a08628c0 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47b8c00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f47a800fa50 tx=0x7f47a8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be7d2bbb1da8337 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f479a000640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8004030 con 0x55b0a08628c0 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47ac032c90 con 0x55b0a08628c0 2024-03-16T12:24:28.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.419+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f47a8018030 con 0x55b0a08628c0 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f47a8034050 con 0x55b0a08628c0 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 10 client.26074 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b0a05a57d0 con 0x55b0a08628c0 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47ac032c90 con 0x55b0a08628c0 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f47a80035f0 con 0x55b0a08628c0 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 10 client.26074 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 10 client.26074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 20 client.26074 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a0878a30 0x55b0a087aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 4 client.26074 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.420+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0a07d4990 con 0x55b0a0878a30 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.421+0000 7f47b3e00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a0878a30 0x55b0a087aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.421+0000 7f47b3e00640 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a0878a30 0x55b0a087aed0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63853ce4f2ae7786 server_cookie=ab608a0d708f8d62 in_seq=0 out_seq=0 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.421+0000 7f479a000640 10 client.26074 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.422+0000 7f479a000640 1 -- 172.21.15.98:0/963776775 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0a07d4990 con 0x55b0a0878a30 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.422+0000 7f479a000640 10 client.26074 handle_command_reply: tid=0 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.422+0000 7f47be262740 1 client.26074 shutdown 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 msgr2=0x55b0a086cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f47a800fa50 tx=0x7f47a8004730 comp rx=0 tx=0).stop 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 shutdown_connections 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a0868420 0x55b0a0874090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a086d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a0878a30 0x55b0a087aed0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 --2- 172.21.15.98:0/963776775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cb10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 >> 172.21.15.98:0/963776775 conn(0x55b0a085de30 msgr2=0x55b0a085d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f479a000640 0 client.26074 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:28.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 shutdown_connections 2024-03-16T12:24:28.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 1 -- 172.21.15.98:0/963776775 wait complete. 2024-03-16T12:24:28.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.423+0000 7f47be262740 20 client.26074 trim_cache size 0 max 16384 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 Processor -- start 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 -- start start 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a087f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41aee20 con 0x55b0a0868420 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41ac6f0 con 0x55b0a08632b0 2024-03-16T12:24:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47be262740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41aa540 con 0x55b0a08628c0 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58564/0 (socket says 172.21.15.98:58564) 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b3e00640 1 -- 172.21.15.98:0/2607136317 learned_addr learned my addr 172.21.15.98:0/2607136317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34558/0 (socket says 172.21.15.98:34558) 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.438+0000 7f47b9600640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a087f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 -- 172.21.15.98:0/2607136317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 msgr2=0x55b0a086cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 -- 172.21.15.98:0/2607136317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 msgr2=0x55b0a087f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a087f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 -- 172.21.15.98:0/2607136317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0a0859c10 con 0x55b0a08632b0 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b8c00640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b9600640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a087f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47a0097c80 tx=0x7f47a00cc7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2c3183581ef4d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47a5200640 1 -- 172.21.15.98:0/2607136317 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a009f8c0 con 0x55b0a08632b0 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47a5200640 1 -- 172.21.15.98:0/2607136317 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47a00a3f10 con 0x55b0a08632b0 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.439+0000 7f47a5200640 1 -- 172.21.15.98:0/2607136317 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a009ac70 con 0x55b0a08632b0 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- 172.21.15.98:0/2607136317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 msgr2=0x55b0a0874050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47a0097c80 tx=0x7f47a00cc7a0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- 172.21.15.98:0/2607136317 shutdown_connections 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a0868420 0x55b0a087f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08632b0 0x55b0a0874050 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 --2- 172.21.15.98:0/2607136317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a08628c0 0x55b0a086cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- 172.21.15.98:0/2607136317 >> 172.21.15.98:0/2607136317 conn(0x55b0a085de30 msgr2=0x55b0a085d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- 172.21.15.98:0/2607136317 shutdown_connections 2024-03-16T12:24:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- 172.21.15.98:0/2607136317 wait complete. 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.440+0000 7f47be262740 1 -- start start 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 Processor -- start 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a0872130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a0868420 0x55b0a0873140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41aabb0 con 0x55b0a08632b0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41ac020 con 0x55b0a08628c0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41ab280 con 0x55b0a0868420 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47be262740 10 client.0 _ll_register_callbacks cb 0x7f47a8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47b3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a0872130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47b8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58574/0 (socket says 172.21.15.98:58574) 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47b8c00640 1 -- 172.21.15.98:0/2699610430 learned_addr learned my addr 172.21.15.98:0/2699610430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.441+0000 7f47b9600640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a0868420 0x55b0a0873140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 -- 172.21.15.98:0/2699610430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a0868420 msgr2=0x55b0a0873140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a0868420 0x55b0a0873140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 -- 172.21.15.98:0/2699610430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 msgr2=0x55b0a0872130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a0872130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 -- 172.21.15.98:0/2699610430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0a0873730 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a0872130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47b8c00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f47a8019040 tx=0x7f47a8003ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa256753f2b3ad01 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f479a000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a803c070 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a00a3f10 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.442+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f47a801e030 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f47a8045050 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 10 client.26090 fetch_fsmap learned FSMap version 24 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b0a08df270 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47a00a3f10 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f47a803f020 con 0x55b0a08628c0 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 10 client.26090 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T12:24:28.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 20 client.26090 populate_metadata read hostname 'smithi098' 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a08dfc40 0x55b0a08e2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 4 client.26090 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 --> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0a07cece0 con 0x55b0a08dfc40 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.443+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a08dfc40 0x55b0a08e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47b3e00640 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a08dfc40 0x55b0a08e2060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a856c63b90887990 server_cookie=65999dd1fa2bc52a in_seq=0 out_seq=0 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f479a000640 10 client.26090 ms_handle_connect on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f479a000640 1 -- 172.21.15.98:0/2699610430 <== mds.0 v2:172.21.15.153:6838/542416413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0a07cece0 con 0x55b0a08dfc40 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f479a000640 10 client.26090 handle_command_reply: tid=0 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 client.26090 shutdown 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 msgr2=0x55b0a0871bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f47a8019040 tx=0x7f47a8003ee0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 shutdown_connections 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:6838/542416413,v1:172.21.15.153:6839/542416413] conn(0x55b0a08dfc40 0x55b0a08e2060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0a0868420 0x55b0a0873140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0a08632b0 0x55b0a0872130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 --2- 172.21.15.98:0/2699610430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b0a08628c0 0x55b0a0871bf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f479a000640 0 client.26090 ms_handle_reset on v2:172.21.15.153:6838/542416413 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 >> 172.21.15.98:0/2699610430 conn(0x55b0a085de30 msgr2=0x55b0a085d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 shutdown_connections 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.444+0000 7f47be262740 1 -- 172.21.15.98:0/2699610430 wait complete. 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47be262740 20 client.26090 trim_cache size 0 max 16384 2024-03-16T12:24:28.428 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 msgr2=0x7f479c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 0x7f479c0a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55b0a085eda0 tx=0x7f47a8005f90 comp rx=0 tx=0).stop 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 msgr2=0x7f47b419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47a000e9f0 tx=0x7f47a000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 shutdown_connections 2024-03-16T12:24:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f479c0a2800 0x7f479c0a4cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47b4101350 0x7f47b41a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47b41009b0 0x7f47b419a1b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 --2- 172.21.15.98:0/1280926886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47b4100050 0x7f47b4199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.445+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 >> 172.21.15.98:0/1280926886 conn(0x7f47b40fbc70 msgr2=0x7f47b40fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.446+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 shutdown_connections 2024-03-16T12:24:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:28.446+0000 7f47bb600640 1 -- 172.21.15.98:0/1280926886 wait complete. 2024-03-16T12:24:29.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:28 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:24:29.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:28 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:24:29.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:28 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:24:29.441 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-16T12:24:29.441 INFO:tasks.fwd_scrub:done joining 2024-03-16T12:24:29.442 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T12:24:29.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.671+0000 7f73be400640 1 -- 172.21.15.98:0/1601771326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8107e00 msgr2=0x7f73b8108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.671+0000 7f73be400640 1 --2- 172.21.15.98:0/1601771326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8107e00 0x7f73b8108220 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009900 tx=0x7f73a802bc30 comp rx=0 tx=0).stop 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 -- 172.21.15.98:0/1601771326 shutdown_connections 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 --2- 172.21.15.98:0/1601771326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b81099b0 0x7f73b8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 --2- 172.21.15.98:0/1601771326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b8108ff0 0x7f73b8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 --2- 172.21.15.98:0/1601771326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8107e00 0x7f73b8108220 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 -- 172.21.15.98:0/1601771326 >> 172.21.15.98:0/1601771326 conn(0x7f73b8076dc0 msgr2=0x7f73b80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 -- 172.21.15.98:0/1601771326 shutdown_connections 2024-03-16T12:24:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.672+0000 7f73be400640 1 -- 172.21.15.98:0/1601771326 wait complete. 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 Processor -- start 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 -- start start 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 0x7f73b819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 0x7f73b819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b81099b0 0x7f73b81a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73b8113a50 con 0x7f73b8107e00 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73b81138d0 con 0x7f73b81099b0 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73be400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73b8113bd0 con 0x7f73b8108ff0 2024-03-16T12:24:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73b7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 0x7f73b819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73b7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 0x7f73b819e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55982/0 (socket says 172.21.15.98:55982) 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 0x7f73b819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.673+0000 7f73b7400640 1 -- 172.21.15.98:0/2128961017 learned_addr learned my addr 172.21.15.98:0/2128961017 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73b7e00640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b81099b0 0x7f73b81a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 -- 172.21.15.98:0/2128961017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b81099b0 msgr2=0x7f73b81a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b81099b0 0x7f73b81a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 -- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 msgr2=0x7f73b819e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 0x7f73b819e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 -- 172.21.15.98:0/2128961017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b81a5d10 con 0x7f73b8108ff0 2024-03-16T12:24:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73afe00640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 0x7f73b819e630 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f73a000e9f0 tx=0x7f73a000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a000ccc0 con 0x7f73b8108ff0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73a00035f0 con 0x7f73b8108ff0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.674+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a000ce60 con 0x7f73b8108ff0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.675+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b81a5fa0 con 0x7f73b8108ff0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.675+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b81a6470 con 0x7f73b8108ff0 2024-03-16T12:24:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.676+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b810f120 con 0x7f73b8108ff0 2024-03-16T12:24:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.678+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f73a00037a0 con 0x7f73b8108ff0 2024-03-16T12:24:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.679+0000 7f73b4c00640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 0x7f739c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.679+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f73a00cefb0 con 0x7f73b8108ff0 2024-03-16T12:24:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.679+0000 7f73b7400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 0x7f739c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.680+0000 7f73b7400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 0x7f739c0a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f73a802c8e0 tx=0x7f73a8005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.681+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f73a009bc90 con 0x7f73b8108ff0 2024-03-16T12:24:29.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.883+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73b802d400 con 0x7f73b8108ff0 2024-03-16T12:24:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.884+0000 7f73b4c00640 1 -- 172.21.15.98:0/2128961017 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f73a0009440 con 0x7f73b8108ff0 2024-03-16T12:24:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:24:29.864 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:24:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 msgr2=0x7f739c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 0x7f739c0a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f73a802c8e0 tx=0x7f73a8005b50 comp rx=0 tx=0).stop 2024-03-16T12:24:29.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 msgr2=0x7f73b819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 0x7f73b819e630 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f73a000e9f0 tx=0x7f73a000edf0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 shutdown_connections 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f739c0a2800 0x7f739c0a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73b81099b0 0x7f73b81a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b8108ff0 0x7f73b819e630 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 --2- 172.21.15.98:0/2128961017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73b8107e00 0x7f73b819e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 >> 172.21.15.98:0/2128961017 conn(0x7f73b8076dc0 msgr2=0x7f73b810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 shutdown_connections 2024-03-16T12:24:29.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:29.887+0000 7f73be400640 1 -- 172.21.15.98:0/2128961017 wait complete. 2024-03-16T12:24:29.885 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:24:29.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.126+0000 7fd3f7600640 1 -- 172.21.15.98:0/3452293875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0104bf0 msgr2=0x7fd3f0105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.126+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3452293875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0104bf0 0x7fd3f0105070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009900 tx=0x7fd3e002bc30 comp rx=0 tx=0).stop 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.126+0000 7fd3f7600640 1 -- 172.21.15.98:0/3452293875 shutdown_connections 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.126+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3452293875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 0x7fd3f010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3452293875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0104bf0 0x7fd3f0105070 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3452293875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0103a00 0x7fd3f0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 -- 172.21.15.98:0/3452293875 >> 172.21.15.98:0/3452293875 conn(0x7fd3f00fef90 msgr2=0x7fd3f01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 -- 172.21.15.98:0/3452293875 shutdown_connections 2024-03-16T12:24:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 -- 172.21.15.98:0/3452293875 wait complete. 2024-03-16T12:24:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.127+0000 7fd3f7600640 1 Processor -- start 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 -- start start 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 0x7fd3f0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 0x7fd3f01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010f6d0 con 0x7fd3f01055b0 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010f550 con 0x7fd3f0104bf0 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f7600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010f850 con 0x7fd3f0103a00 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 0x7fd3f0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58604/0 (socket says 172.21.15.98:58604) 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3e7e00640 1 -- 172.21.15.98:0/3144987224 learned_addr learned my addr 172.21.15.98:0/3144987224 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.128+0000 7fd3f5600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 0x7fd3f01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 -- 172.21.15.98:0/3144987224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 msgr2=0x7fd3f0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 0x7fd3f0199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 -- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 msgr2=0x7fd3f01a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 0x7fd3f01a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 -- 172.21.15.98:0/3144987224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f01a1960 con 0x7fd3f0104bf0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3f4c00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 0x7fd3f0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3e7e00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0000c00 tx=0x7fd3e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0043020 con 0x7fd3f0104bf0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e00054e0 con 0x7fd3f0104bf0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.129+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f01a1bf0 con 0x7fd3f0104bf0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.130+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0003c40 con 0x7fd3f0104bf0 2024-03-16T12:24:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.130+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f01a2190 con 0x7fd3f0104bf0 2024-03-16T12:24:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.131+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f010ad20 con 0x7fd3f0104bf0 2024-03-16T12:24:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.133+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3e0046020 con 0x7fd3f0104bf0 2024-03-16T12:24:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.134+0000 7fd3ee000640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.134+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3e00f1190 con 0x7fd3f0104bf0 2024-03-16T12:24:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.134+0000 7fd3f4c00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.135+0000 7fd3f4c00640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 0x7fd3d00a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0104a50 tx=0x7fd3d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:30.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.141+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3e004d020 con 0x7fd3f0104bf0 2024-03-16T12:24:30.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:30 smithi098 ceph-mon[31633]: pgmap v494: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-03-16T12:24:30.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:24:30.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:24:30.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:30 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:24:30.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:30 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2128961017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:24:30.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:30 smithi153 ceph-mon[36901]: pgmap v494: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-03-16T12:24:30.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:24:30.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:24:30.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:30 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:24:30.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:30 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2128961017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:24:30.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:30 smithi173 ceph-mon[36632]: pgmap v494: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 13 op/s 2024-03-16T12:24:30.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:24:30.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:24:30.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:30 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' 2024-03-16T12:24:30.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:30 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2128961017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:24:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.342+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd3f002d340 con 0x7fd3f0104bf0 2024-03-16T12:24:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.342+0000 7fd3ee000640 1 -- 172.21.15.98:0/3144987224 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fd3e00bde70 con 0x7fd3f0104bf0 2024-03-16T12:24:30.322 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:24:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 msgr2=0x7fd3d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 0x7fd3d00a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0104a50 tx=0x7fd3d8005f90 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 msgr2=0x7fd3f019a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0000c00 tx=0x7fd3e002c650 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 shutdown_connections 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f01055b0 0x7fd3f01a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f0104bf0 0x7fd3f019a280 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 --2- 172.21.15.98:0/3144987224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3f0103a00 0x7fd3f0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 >> 172.21.15.98:0/3144987224 conn(0x7fd3f00fef90 msgr2=0x7fd3f0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 shutdown_connections 2024-03-16T12:24:30.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.345+0000 7fd3f7600640 1 -- 172.21.15.98:0/3144987224 wait complete. 2024-03-16T12:24:30.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.585+0000 7f6d7dc00640 1 -- 172.21.15.98:0/547772261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 msgr2=0x7f6d78109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.585+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/547772261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d78109470 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009900 tx=0x7f6d6002bc40 comp rx=0 tx=0).stop 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 -- 172.21.15.98:0/547772261 shutdown_connections 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/547772261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 0x7f6d78110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/547772261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d78109470 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/547772261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d78108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 -- 172.21.15.98:0/547772261 >> 172.21.15.98:0/547772261 conn(0x7f6d78076da0 msgr2=0x7f6d780791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 -- 172.21.15.98:0/547772261 shutdown_connections 2024-03-16T12:24:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.586+0000 7f6d7dc00640 1 -- 172.21.15.98:0/547772261 wait complete. 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.587+0000 7f6d7dc00640 1 Processor -- start 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.587+0000 7f6d7dc00640 1 -- start start 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d7819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 0x7f6d781a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78113a80 con 0x7f6d781099b0 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78113900 con 0x7f6d78108ff0 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d7dc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78113c00 con 0x7f6d78107e00 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34612/0 (socket says 172.21.15.98:34612) 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 -- 172.21.15.98:0/4261424085 learned_addr learned my addr 172.21.15.98:0/4261424085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d6e000640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d7819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d77400640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 0x7f6d781a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 -- 172.21.15.98:0/4261424085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 msgr2=0x7f6d7819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d7819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 -- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 msgr2=0x7f6d781a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 0x7f6d781a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.588+0000 7f6d76a00640 1 -- 172.21.15.98:0/4261424085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d781a5d40 con 0x7f6d78107e00 2024-03-16T12:24:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d76a00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800d950 tx=0x7f6d6800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68017070 con 0x7f6d78107e00 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d68009c30 con 0x7f6d78107e00 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d781a5fd0 con 0x7f6d78107e00 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6800f600 con 0x7f6d78107e00 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.589+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d781a6440 con 0x7f6d78107e00 2024-03-16T12:24:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.590+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d7810f120 con 0x7f6d78107e00 2024-03-16T12:24:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.592+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d6801a070 con 0x7f6d78107e00 2024-03-16T12:24:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.593+0000 7f6d6fe00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 0x7f6d5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.593+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6d680ce7c0 con 0x7f6d78107e00 2024-03-16T12:24:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.594+0000 7f6d6e000640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 0x7f6d5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.594+0000 7f6d6e000640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 0x7f6d5c0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d6002c7c0 tx=0x7f6d60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.596+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d68009de0 con 0x7f6d78107e00 2024-03-16T12:24:30.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.793+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6d78062410 con 0x7f6d78107e00 2024-03-16T12:24:30.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.794+0000 7f6d6fe00640 1 -- 172.21.15.98:0/4261424085 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f6d6809ae50 con 0x7f6d78107e00 2024-03-16T12:24:30.773 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:24:30.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 msgr2=0x7f6d5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 0x7f6d5c0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6d6002c7c0 tx=0x7f6d60005fd0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 msgr2=0x7f6d7819e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800d950 tx=0x7f6d6800de10 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 shutdown_connections 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6d5c0a2800 0x7f6d5c0a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d781099b0 0x7f6d781a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d78108ff0 0x7f6d7819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 --2- 172.21.15.98:0/4261424085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d78107e00 0x7f6d7819e120 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 >> 172.21.15.98:0/4261424085 conn(0x7f6d78076da0 msgr2=0x7f6d7810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.796+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 shutdown_connections 2024-03-16T12:24:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:30.797+0000 7f6d7dc00640 1 -- 172.21.15.98:0/4261424085 wait complete. 2024-03-16T12:24:30.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:24:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 -- 172.21.15.98:0/110185289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad981054b0 msgr2=0x7fad9810bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 --2- 172.21.15.98:0/110185289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad981054b0 0x7fad9810bd40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad90009be0 tx=0x7fad9002bef0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 -- 172.21.15.98:0/110185289 shutdown_connections 2024-03-16T12:24:31.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 --2- 172.21.15.98:0/110185289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad981054b0 0x7fad9810bd40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 --2- 172.21.15.98:0/110185289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad98104af0 0x7fad98104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 --2- 172.21.15.98:0/110185289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98103900 0x7fad98103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 -- 172.21.15.98:0/110185289 >> 172.21.15.98:0/110185289 conn(0x7fad980fee90 msgr2=0x7fad981012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 -- 172.21.15.98:0/110185289 shutdown_connections 2024-03-16T12:24:31.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.035+0000 7fad9d600640 1 -- 172.21.15.98:0/110185289 wait complete. 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 Processor -- start 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 -- start start 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 0x7fad9819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 0x7fad981a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad9810f5b0 con 0x7fad98104af0 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad9810f430 con 0x7fad98103900 2024-03-16T12:24:31.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9d600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fad9810f730 con 0x7fad981054b0 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad8f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 0x7fad9819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad9cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 0x7fad981a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58650/0 (socket says 172.21.15.98:58650) 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.036+0000 7fad97e00640 1 -- 172.21.15.98:0/2662760820 learned_addr learned my addr 172.21.15.98:0/2662760820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 -- 172.21.15.98:0/2662760820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 msgr2=0x7fad981a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 0x7fad981a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 -- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 msgr2=0x7fad9819a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 0x7fad9819a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 -- 172.21.15.98:0/2662760820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad981a1780 con 0x7fad98103900 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad9cc00640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 0x7fad981a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad8f400640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 0x7fad9819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:24:31.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad97e00640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad8800e9c0 tx=0x7fad8800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8800cd80 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad880035f0 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad981a1a70 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.037+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88010690 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.038+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad981a2010 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.039+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad9810ab50 con 0x7fad98103900 2024-03-16T12:24:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.040+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad880037a0 con 0x7fad98103900 2024-03-16T12:24:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.042+0000 7fad95600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 0x7fad7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.042+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fad880cf0f0 con 0x7fad98103900 2024-03-16T12:24:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.042+0000 7fad8f400640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 0x7fad7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.042+0000 7fad8f400640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 0x7fad7c0a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fad800096c0 tx=0x7fad80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.050+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad8809be30 con 0x7fad98103900 2024-03-16T12:24:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.251+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fad9802d1b0 con 0x7fad98103900 2024-03-16T12:24:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.252+0000 7fad95600640 1 -- 172.21.15.98:0/2662760820 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fad8809b730 con 0x7fad98103900 2024-03-16T12:24:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 msgr2=0x7fad7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 0x7fad7c0a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fad800096c0 tx=0x7fad80009260 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 msgr2=0x7fad98199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad8800e9c0 tx=0x7fad8800ee80 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 shutdown_connections 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fad7c0a2880 0x7fad7c0a4d40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad981054b0 0x7fad981a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad98104af0 0x7fad9819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 --2- 172.21.15.98:0/2662760820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad98103900 0x7fad98199b60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 >> 172.21.15.98:0/2662760820 conn(0x7fad980fee90 msgr2=0x7fad98107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 shutdown_connections 2024-03-16T12:24:31.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.255+0000 7fad9d600640 1 -- 172.21.15.98:0/2662760820 wait complete. 2024-03-16T12:24:31.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:31 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3144987224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:31 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4261424085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:31 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3144987224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:31 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4261424085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:24:31.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:31 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3144987224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:31 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4261424085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:24:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 -- 172.21.15.98:0/2818143422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 msgr2=0x7f0bdc0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 --2- 172.21.15.98:0/2818143422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc0746c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009860 tx=0x7f0bc402bb70 comp rx=0 tx=0).stop 2024-03-16T12:24:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 -- 172.21.15.98:0/2818143422 shutdown_connections 2024-03-16T12:24:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 --2- 172.21.15.98:0/2818143422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 0x7f0bdc10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 --2- 172.21.15.98:0/2818143422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc0746c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 --2- 172.21.15.98:0/2818143422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 -- 172.21.15.98:0/2818143422 >> 172.21.15.98:0/2818143422 conn(0x7f0bdc0fcd60 msgr2=0x7f0bdc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 -- 172.21.15.98:0/2818143422 shutdown_connections 2024-03-16T12:24:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.494+0000 7f0be2a00640 1 -- 172.21.15.98:0/2818143422 wait complete. 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 Processor -- start 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 -- start start 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 0x7f0bdc1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10f630 con 0x7f0bdc074240 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10f4b0 con 0x7f0bdc1096b0 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.495+0000 7f0be2a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10f7b0 con 0x7f0bdc075820 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34628/0 (socket says 172.21.15.98:34628) 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 -- 172.21.15.98:0/1083797466 learned_addr learned my addr 172.21.15.98:0/1083797466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0be0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 0x7f0bdc1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 -- 172.21.15.98:0/1083797466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 msgr2=0x7f0bdc1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 0x7f0bdc1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 -- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 msgr2=0x7f0bdc199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 -- 172.21.15.98:0/1083797466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc1a1930 con 0x7f0bdc075820 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.496+0000 7f0bdb400640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40349e0 tx=0x7f0bc4032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.497+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4041030 con 0x7f0bdc075820 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.497+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc402c510 con 0x7f0bdc075820 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.497+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc1a1bc0 con 0x7f0bdc075820 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.497+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4039040 con 0x7f0bdc075820 2024-03-16T12:24:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.497+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1a2100 con 0x7f0bdc075820 2024-03-16T12:24:31.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.498+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc1083e0 con 0x7f0bdc075820 2024-03-16T12:24:31.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.500+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0bc4044070 con 0x7f0bdc075820 2024-03-16T12:24:31.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.501+0000 7f0bd8c00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 0x7f0bc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.501+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0bc40f0300 con 0x7f0bdc075820 2024-03-16T12:24:31.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.501+0000 7f0bdbe00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 0x7f0bc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.502+0000 7f0bdbe00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 0x7f0bc00a4d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc19b1f0 tx=0x7f0bcc010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.503+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bc40bd060 con 0x7f0bdc075820 2024-03-16T12:24:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.701+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0bdc02d400 con 0x7f0bdc075820 2024-03-16T12:24:31.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.701+0000 7f0bd8c00640 1 -- 172.21.15.98:0/1083797466 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f0bc40bc960 con 0x7f0bdc075820 2024-03-16T12:24:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 msgr2=0x7f0bc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 0x7f0bc00a4d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc19b1f0 tx=0x7f0bcc010040 comp rx=0 tx=0).stop 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 msgr2=0x7f0bdc19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40349e0 tx=0x7f0bc4032450 comp rx=0 tx=0).stop 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 shutdown_connections 2024-03-16T12:24:31.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f0bc00a2880 0x7f0bc00a4d40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bdc1096b0 0x7f0bdc1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bdc075820 0x7f0bdc19a260 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 --2- 172.21.15.98:0/1083797466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bdc074240 0x7f0bdc199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 >> 172.21.15.98:0/1083797466 conn(0x7f0bdc0fcd60 msgr2=0x7f0bdc1055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 shutdown_connections 2024-03-16T12:24:31.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.704+0000 7f0be2a00640 1 -- 172.21.15.98:0/1083797466 wait complete. 2024-03-16T12:24:31.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 -- 172.21.15.98:0/3877024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8107e20 msgr2=0x7f72a8108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 --2- 172.21.15.98:0/3877024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8107e20 0x7f72a8108240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7298009900 tx=0x7f729802bc40 comp rx=0 tx=0).stop 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 -- 172.21.15.98:0/3877024654 shutdown_connections 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 --2- 172.21.15.98:0/3877024654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a8110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 --2- 172.21.15.98:0/3877024654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8108f70 0x7f72a81093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.934+0000 7f72aee00640 1 --2- 172.21.15.98:0/3877024654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8107e20 0x7f72a8108240 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.935+0000 7f72aee00640 1 -- 172.21.15.98:0/3877024654 >> 172.21.15.98:0/3877024654 conn(0x7f72a8076df0 msgr2=0x7f72a8079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.935+0000 7f72aee00640 1 -- 172.21.15.98:0/3877024654 shutdown_connections 2024-03-16T12:24:31.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.935+0000 7f72aee00640 1 -- 172.21.15.98:0/3877024654 wait complete. 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 Processor -- start 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 -- start start 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8107e20 0x7f72a819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8108f70 0x7f72a819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8113b10 con 0x7f72a8107e20 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8113990 con 0x7f72a8109930 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.936+0000 7f72aee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72a8113c90 con 0x7f72a8108f70 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:58682/0 (socket says 172.21.15.98:58682) 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 -- 172.21.15.98:0/3290156858 learned_addr learned my addr 172.21.15.98:0/3290156858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72a7400640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8108f70 0x7f72a819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72a7e00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8107e20 0x7f72a819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 -- 172.21.15.98:0/3290156858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8108f70 msgr2=0x7f72a819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8108f70 0x7f72a819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 -- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8107e20 msgr2=0x7f72a819e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8107e20 0x7f72a819e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 -- 172.21.15.98:0/3290156858 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a81a5ce0 con 0x7f72a8109930 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.937+0000 7f72ace00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f72a000eb10 tx=0x7f72a000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.938+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0004b20 con 0x7f72a8109930 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.938+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a0004cc0 con 0x7f72a8109930 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.938+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81a5fd0 con 0x7f72a8109930 2024-03-16T12:24:31.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.938+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0010760 con 0x7f72a8109930 2024-03-16T12:24:31.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.938+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81a6490 con 0x7f72a8109930 2024-03-16T12:24:31.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.939+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a810f0d0 con 0x7f72a8109930 2024-03-16T12:24:31.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.941+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72a0005020 con 0x7f72a8109930 2024-03-16T12:24:31.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.943+0000 7f72a4c00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:24:31.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.943+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72a00cf590 con 0x7f72a8109930 2024-03-16T12:24:31.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.943+0000 7f72a7e00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:24:31.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.944+0000 7f72a7e00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 0x7f728c0a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f729802c250 tx=0x7f7298005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:24:31.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:31.946+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72a009c270 con 0x7f72a8109930 2024-03-16T12:24:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.200+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f72a8062480 con 0x7f72a8109930 2024-03-16T12:24:32.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.210+0000 7f72a4c00640 1 -- 172.21.15.98:0/3290156858 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739830 (secure 0 0 0) 0x7f72a009bb70 con 0x7f72a8109930 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 18, 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 14152, 2024-03-16T12:24:32.202 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3089653509 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3089653509 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.203 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/3089653509", 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-03-16T12:10:42.633257+0000", 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24109, 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-16T12:24:32.204 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.206 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-16T12:24:32.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-16T12:24:32.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-16T12:24:32.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-16T12:24:32.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-16T12:24:32.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:24:32.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.234 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.239 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:24:32.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-03-16T12:24:32.246 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:24:32.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:24:32.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:24:32.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.271 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.277 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-16T12:24:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-16T12:24:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-16T12:24:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-16T12:24:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-16T12:24:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-16T12:24:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-16T12:24:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:24:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:24:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-16T12:24:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-16T12:24:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-16T12:24:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-16T12:24:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-16T12:24:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.400 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-16T12:24:32.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-16T12:24:32.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-16T12:24:32.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-16T12:24:32.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-16T12:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.420 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.422 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.424 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.425 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.434 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.435 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.437 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.439 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:24:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:24:32.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:24:32.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:24:32.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:24:32.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.452 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.455 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:32.465 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.467 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.482 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.485 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-16T12:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-16T12:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-16T12:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-16T12:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.611 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-16T12:24:32.616 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.617 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.620 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.623 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.625 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-16T12:24:32.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-03-16T12:24:32.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.629 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-16T12:24:32.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.638 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-16T12:24:32.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.643 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24110, 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.648 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-16T12:24:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.657 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-16T12:24:32.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-16T12:24:32.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.666 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.668 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.671 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-03-16T12:24:32.675 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.680 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:24:32.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:24:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-03-16T12:24:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-16T12:24:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-16T12:24:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-16T12:24:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:24:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-16T12:24:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:24:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-16T12:24:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-16T12:24:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.731 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.731 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:24:32.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:24:32.735 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.736 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-16T12:24:32.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.741 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.744 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-16T12:24:32.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.748 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:24:32.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-16T12:24:32.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.767 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.782 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.786 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:24:32.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:24:32.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.798 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:24:32.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.805 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.814 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.822 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:32.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.825 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.830 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:24:32.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.837 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.840 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.841 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.846 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:32.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:24:32.848 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:24:32.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.855 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.856 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.858 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.859 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.860 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:24:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:32.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.868 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.872 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-16T12:24:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:24:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-16T12:24:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-16T12:24:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:24:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-16T12:24:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.902 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.904 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.908 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.917 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.920 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:24:32.921 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.922 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.924 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.926 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-16T12:24:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.930 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-16T12:24:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-16T12:24:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-16T12:24:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:24:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:24:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-16T12:24:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-16T12:24:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-16T12:24:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-16T12:24:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-16T12:24:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-16T12:24:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-16T12:24:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-16T12:24:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-16T12:24:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-16T12:24:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.029 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-16T12:24:33.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.036 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-16T12:24:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-16T12:24:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-16T12:24:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-16T12:24:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-16T12:24:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.103 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.108 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.110 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.114 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-16T12:24:33.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "restful" 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.117 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.119 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-03-16T12:24:33.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-03-16T12:24:33.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-03-16T12:24:33.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:24:33.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:24:33.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.142 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.143 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.145 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:24:33.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.154 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:24:33.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.158 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:33.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-03-16T12:24:33.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:24:33.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-03-16T12:24:33.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.172 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.179 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-03-16T12:24:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-03-16T12:24:33.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.196 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.198 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.207 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-03-16T12:24:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:24:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:24:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:24:33.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:24:33.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:24:33.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:24:33.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-03-16T12:24:33.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:24:33.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:24:33.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.256 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:24:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:24:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-03-16T12:24:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:24:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:24:33.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:24:33.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-03-16T12:24:33.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-03-16T12:24:33.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.286 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.287 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.288 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.296 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:33.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:33.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:33.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.306 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.308 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-03-16T12:24:33.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-03-16T12:24:33.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-03-16T12:24:33.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.318 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:33.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-03-16T12:24:33.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-03-16T12:24:33.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.330 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.331 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.332 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.335 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.337 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:24:33.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:24:33.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-03-16T12:24:33.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.354 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.362 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-03-16T12:24:33.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.377 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-03-16T12:24:33.384 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-03-16T12:24:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-03-16T12:24:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-03-16T12:24:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-03-16T12:24:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-03-16T12:24:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-03-16T12:24:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-03-16T12:24:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-03-16T12:24:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-03-16T12:24:33.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.508 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-03-16T12:24:33.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.517 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.520 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.521 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.527 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-03-16T12:24:33.529 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.530 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-03-16T12:24:33.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.538 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.541 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-03-16T12:24:33.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-03-16T12:24:33.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-03-16T12:24:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-03-16T12:24:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.582 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.584 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-03-16T12:24:33.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-03-16T12:24:33.592 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-03-16T12:24:33.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-03-16T12:24:33.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-03-16T12:24:33.597 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.598 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-16T12:24:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-16T12:24:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "squid": [ 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-03-16T12:24:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 3, 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-16T12:24:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3627906864 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 538510969 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4287586498 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2698332152 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-16T12:24:33.606 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:32 smithi173 ceph-mon[36632]: pgmap v495: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:32 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2662760820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:32 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1083797466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:33 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3290156858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:32 smithi153 ceph-mon[36901]: pgmap v495: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:32 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2662760820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:32 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1083797466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.609 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:33 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3290156858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 msgr2=0x7f728c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 0x7f728c0a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f729802c250 tx=0x7f7298005fd0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 msgr2=0x7f72a81a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f72a000eb10 tx=0x7f72a000c2d0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 shutdown_connections 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72a8109930 0x7f72a81a5670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8108f70 0x7f72a819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 --2- 172.21.15.98:0/3290156858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72a8107e20 0x7f72a819e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:24:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 >> 172.21.15.98:0/3290156858 conn(0x7f72a8076df0 msgr2=0x7f72a810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:24:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 shutdown_connections 2024-03-16T12:24:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:24:32.220+0000 7f72aee00640 1 -- 172.21.15.98:0/3290156858 wait complete. 2024-03-16T12:24:33.617 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T12:24:33.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T12:24:33.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:32 smithi098 ceph-mon[31633]: pgmap v495: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T12:24:33.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:32 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2662760820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.619 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:32 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1083797466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:24:33.619 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:33 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3290156858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:24:34.078 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:24:34.280 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:34 smithi098 ceph-mon[31633]: pgmap v496: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T12:24:34.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:34 smithi173 ceph-mon[36632]: pgmap v496: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T12:24:34.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:34 smithi153 ceph-mon[36901]: pgmap v496: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-16T12:24:36.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:36 smithi098 ceph-mon[31633]: pgmap v497: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:36.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:36 smithi153 ceph-mon[36901]: pgmap v497: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:36.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:36 smithi173 ceph-mon[36632]: pgmap v497: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:36.540 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.003812601, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000640301, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 5, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 78, 2024-03-16T12:24:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3184, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 91536, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.015662839, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011945011, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002092157, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 77, 2024-03-16T12:24:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008957505, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000116331 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-16T12:24:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 91536 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 104, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 101, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 443824, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 61328, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.040884159, 2024-03-16T12:24:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.014902327, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.028926778, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.004216852, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 100, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008999085, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000089990 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 443824, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 61328 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 149854, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 28, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T12:24:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 105, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 429183, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 105, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 105, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 429183, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:36.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:37.615 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-16T12:24:37.615 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-16T12:24:37.615 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-16T12:24:37.615 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-16T12:24:37.615 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T12:24:37.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T12:24:38.049 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:24:38.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:38 smithi173 ceph-mon[36632]: pgmap v498: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:38.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:38 smithi098 ceph-mon[31633]: pgmap v498: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:38.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:38 smithi153 ceph-mon[36901]: pgmap v498: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:40.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:40 smithi098 ceph-mon[31633]: pgmap v499: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:40.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:40 smithi173 ceph-mon[36632]: pgmap v499: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:40.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:40 smithi153 ceph-mon[36901]: pgmap v499: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:24:40.512 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.003821236, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001047990, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:40.513 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 115, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 110, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 448752, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 66256, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.039001810, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.012544688, 2024-03-16T12:24:40.514 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.028109277, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.004451128, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 110, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.011924183, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000108401 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 448752, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 66256 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 82, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 160, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 92144, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.017314507, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011890702, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.003588126, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 80, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.006514114, 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000081426 2024-03-16T12:24:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 92144 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T12:24:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148823, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 27, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 112, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 430093, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 112, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 112, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 430093, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:40.523 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:41.444 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T12:24:41.445 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T12:24:41.445 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-16T12:24:41.445 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-16T12:24:41.445 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T12:24:41.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T12:24:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:24:42.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:42 smithi098 ceph-mon[31633]: pgmap v500: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:42.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:42 smithi173 ceph-mon[36632]: pgmap v500: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:42.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:42 smithi153 ceph-mon[36901]: pgmap v500: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-16T12:24:44.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:44 smithi098 ceph-mon[31633]: pgmap v501: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:44.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:44 smithi173 ceph-mon[36632]: pgmap v501: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:44.402 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 4, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3024, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 448, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.004697847, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000086575, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001733639, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3, 2024-03-16T12:24:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.001836624, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000612208 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 79, 2024-03-16T12:24:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 160, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 91616, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.016217384, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011018423, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.003344485, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 77, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.006683213, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000086794 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 91616 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 107, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 104, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 444640, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 63248, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.040307275, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.016249116, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.024897934, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.004156245, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 103, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.009581912, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000093028 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 444640, 2024-03-16T12:24:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 63248 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:44.410 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148823, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 27, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:44.411 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 108, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 429573, 2024-03-16T12:24:44.412 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 108, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 108, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 429573, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:44.413 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:44.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:44 smithi153 ceph-mon[36901]: pgmap v501: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-16T12:24:45.352 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-16T12:24:45.352 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-16T12:24:45.352 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-16T12:24:45.352 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-16T12:24:45.352 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T12:24:45.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T12:24:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:24:46.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:46 smithi098 ceph-mon[31633]: pgmap v502: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:24:46.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:46 smithi173 ceph-mon[36632]: pgmap v502: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:24:46.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:46 smithi153 ceph-mon[36901]: pgmap v502: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:24:48.156 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 113, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 110, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 446272, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 67088, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.047817262, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.016784543, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.029765862, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.004315293, 2024-03-16T12:24:48.157 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 109, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012496643, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000114648 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 446272, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 67088 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 5, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 86, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3184, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 92944, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 3, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.015965842, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011845451, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002321403, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 85, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008779690, 2024-03-16T12:24:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000103290 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 92944 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-16T12:24:48.160 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000955937, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000813602, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:48.161 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:48.162 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:48.163 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:48.164 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 150885, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-16T12:24:48.165 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-16T12:24:48.166 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 114, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 430353, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 114, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 114, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 430353, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-16T12:24:48.167 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-16T12:24:48.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:48 smithi098 ceph-mon[31633]: pgmap v503: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:24:48.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:48 smithi173 ceph-mon[36632]: pgmap v503: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:24:48.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:48 smithi153 ceph-mon[36901]: pgmap v503: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:24:49.073 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T12:24:49.073 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T12:24:49.073 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-16T12:24:49.073 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-16T12:24:49.073 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T12:24:49.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T12:24:49.326 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:24:50.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:50 smithi173 ceph-mon[36632]: pgmap v504: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:24:50.470 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:50 smithi153 ceph-mon[36901]: pgmap v504: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:24:50.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:50 smithi098 ceph-mon[31633]: pgmap v504: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:24:51.206 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 703, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 680, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 188320, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 1284091, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 49, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.068275079, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.024149197, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.023790643, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.012631373, 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:51.207 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 678, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.119699901, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001651474 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 519, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000079029, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000152 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 687, 2024-03-16T12:24:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 546, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 252535, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 450745, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 14, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.049291249, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.013892761, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.022373266, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.013810720, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 545, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.639189486, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001172824 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.209 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 610, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000121614, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000199 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 187664, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 318800 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 729, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 683, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 643040, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 271753, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 10, 2024-03-16T12:24:51.210 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.052036321, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.015639414, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.029396221, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.011862181, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 682, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.148117053, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000217180 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 532, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000090460, 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000170 2024-03-16T12:24:51.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 487872, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 75376 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 366, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.106379451, 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000290654 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 140, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004458486, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000031846 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 66, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 66, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 66, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.075739031, 2024-03-16T12:24:51.213 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001147561 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 12, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 24, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 24, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 23, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-16T12:24:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 1, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 1, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 2, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 100, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 92, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 3, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T12:24:51.215 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 70, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 70, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 42, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 5, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 37, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 28, 2024-03-16T12:24:51.216 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 14, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 30, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 104, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 84, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 1, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 27 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 8, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-16T12:24:51.217 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 16, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 6, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 6, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:24:51.218 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 5, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 160, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 160, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T12:24:51.219 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4805760, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 130, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.218725386, 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001682502 2024-03-16T12:24:51.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 23, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 34, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 11, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 18, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 20, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 2, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 24, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 35, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 11, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 1, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 19, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 18, 2024-03-16T12:24:51.221 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 53524, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 231804 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 89, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 66, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 14, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.007084359, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000885544 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T12:24:51.222 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 29, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.006943159, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000239419 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T12:24:51.223 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T12:24:51.224 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T12:24:51.225 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.226 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.226 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.226 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T12:24:51.226 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.013326909, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001665863 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000143834, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000143834 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004485772, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002242886 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002840746, 2024-03-16T12:24:51.228 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001420373 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.015984101, 2024-03-16T12:24:51.229 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.007992050 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:51.230 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001103616, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000551808 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.231 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.018693939, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002336742 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 4, 2024-03-16T12:24:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.005132596, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001283149 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 3, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 2, 2024-03-16T12:24:51.233 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 13, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 13, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 196, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:51.234 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:51.235 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 307955, 2024-03-16T12:24:51.236 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 127, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 440, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 5, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 8696, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 166, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T12:24:51.237 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 95106, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 660, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 927, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 628516, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 31, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 927, 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T12:24:51.238 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 927, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 24.400578125, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.026322090 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 927, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 973 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 927, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 81, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 846, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-03-16T12:24:51.239 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 11, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 6, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 131, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 21, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 650, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 75, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 22, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:24:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T12:24:51.241 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 91, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 7, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 12, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 12, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T12:24:51.242 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 47, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 10 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.243 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 137, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 5, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 138, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 6, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2115, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 862867, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:51.244 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2115, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2115, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 862867, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:51.245 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 927, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 629154, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 927, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 629154, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.246 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 927, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 927, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 927, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 927, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.247 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 6, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 638, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 6, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 638, 2024-03-16T12:24:51.248 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x565210f6a500": { 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 160, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 611456, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 160, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:51.249 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 130, 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 611456, 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:51.250 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T12:24:51.901 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-16T12:24:51.901 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-16T12:24:51.901 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=8 2024-03-16T12:24:51.901 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-16T12:24:51.901 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T12:24:51.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T12:24:52.159 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:24:52.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:52 smithi153 ceph-mon[36901]: pgmap v505: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:24:52.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:52 smithi173 ceph-mon[36632]: pgmap v505: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:24:52.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:52 smithi098 ceph-mon[31633]: pgmap v505: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:24:53.939 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 122, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 118, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 466418, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 68596, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.015510929, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.004228367, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.011269966, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001871300, 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:53.940 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 117, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004246885, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000036298 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 465584, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 68272 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 205, 2024-03-16T12:24:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 289, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 50280, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 357871, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.017869796, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.006534171, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.004275941, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005312119, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 287, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.011269357, 2024-03-16T12:24:53.942 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000039266 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 293312 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 700, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 700, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 211913, 2024-03-16T12:24:53.943 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 221834, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.045395118, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.014842663, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.012327445, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.014633991, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 699, 2024-03-16T12:24:53.944 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.016456878, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000023543 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-16T12:24:53.945 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1592, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.031332363, 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000019681 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T12:24:53.947 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T12:24:53.948 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T12:24:53.949 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:24:53.950 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-16T12:24:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T12:24:53.952 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:24:53.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-03-16T12:24:53.954 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 3, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195202, 2024-03-16T12:24:53.955 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4194892, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 3 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-03-16T12:24:53.956 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 46264, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215420 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-03-16T12:24:53.957 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.958 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.959 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.960 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T12:24:53.961 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.962 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.962 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.962 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.965 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T12:24:53.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T12:24:53.967 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.968 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 1, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 0, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T12:24:53.969 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 1733, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:53.970 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:53.971 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 190001, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 50, 2024-03-16T12:24:53.972 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 8696, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 166, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8464, 2024-03-16T12:24:53.973 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 900, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 900, 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T12:24:53.974 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 900, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.618989323, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000687765 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 900, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 901 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 900, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 900, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T12:24:53.975 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 547, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 351, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-03-16T12:24:53.976 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T12:24:53.977 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 91, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 2, 2024-03-16T12:24:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 5, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 5, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.979 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:53.980 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 1023, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 627285, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 1023, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 1023, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 627285, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 900, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 903, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 900, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 903, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.982 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 900, 2024-03-16T12:24:53.983 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 900, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 900, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 900, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:53.984 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55bd07fa86e0": { 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:53.985 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.986 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:53.987 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T12:24:54.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:54 smithi153 ceph-mon[36901]: pgmap v506: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:54.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:54 smithi173 ceph-mon[36632]: pgmap v506: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:54.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:54 smithi098 ceph-mon[31633]: pgmap v506: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:54.794 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T12:24:54.795 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T12:24:54.795 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-16T12:24:54.795 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-16T12:24:54.795 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T12:24:54.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T12:24:55.048 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:24:56.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:56 smithi153 ceph-mon[36901]: pgmap v507: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:24:56.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:56 smithi173 ceph-mon[36632]: pgmap v507: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:24:56.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:56 smithi098 ceph-mon[31633]: pgmap v507: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-16T12:24:56.800 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 256, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 279, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 75784, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 88452, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 8, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.023173159, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.009745110, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.010942532, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.001325732, 2024-03-16T12:24:56.801 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 282, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.027708728, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000098257 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 486, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000080297, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000165 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:56.802 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 179, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 174, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 484949, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 93254, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 5, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.017261972, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.005063616, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.013201432, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.002098091, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 173, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.013255223, 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000076619 2024-03-16T12:24:56.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 25, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000003625, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000145 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 472240, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 71952 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 25, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 111, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 5351, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 381332, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 22, 2024-03-16T12:24:56.804 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 7, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.009970109, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.003320813, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.005585309, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000109384, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 105, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.007561755, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000072016 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 23, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000002625, 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000114 2024-03-16T12:24:56.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1936, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 216032 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 51, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001009295, 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000019790 2024-03-16T12:24:56.806 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000005918, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000002959 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 2, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 1, 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-03-16T12:24:56.807 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000137897, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000137897 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 1, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 11, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 16, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 16, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-03-16T12:24:56.808 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 13, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 4, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 1, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 1, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 3, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 8, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 4, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 1, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 3, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-03-16T12:24:56.809 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 4, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 234, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:24:56.810 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-03-16T12:24:56.811 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 11, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 7, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 8, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:24:56.812 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 3, 2024-03-16T12:24:56.813 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 3, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 2, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4195202, 2024-03-16T12:24:56.814 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002731245, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001365622 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 20, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 22, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 2, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 19, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 22, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 3, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 16, 2024-03-16T12:24:56.815 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 20, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 4, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 1, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 1, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 48048, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 223612 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 2, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 2, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 3, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-03-16T12:24:56.816 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-03-16T12:24:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.818 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000137897, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000137897 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.819 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-03-16T12:24:56.820 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-03-16T12:24:56.821 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.823 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.824 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.825 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-03-16T12:24:56.827 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 1, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 1, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 0, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 1, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 1, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 1, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 202, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:56.828 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:56.829 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:56.830 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 247431, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 82, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:56.831 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 8696, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 166, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 80997, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 327, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 38, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 11321, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 30, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 38, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 38, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.315994578, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.008315646 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-03-16T12:24:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 38, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 61 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 38, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 0, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 38, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 10, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 0, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 2, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 6, 2024-03-16T12:24:56.834 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 6, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 20, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-03-16T12:24:56.835 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-03-16T12:24:56.836 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 91, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 2, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 8, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 8, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-03-16T12:24:56.837 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 16, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 1 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 456, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 514360, 2024-03-16T12:24:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 456, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 456, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 514360, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 38, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 11321, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 38, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 11321, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 38, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 38, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 38, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 38, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55e1cb9d2500": { 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 3, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 898, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 3, 2024-03-16T12:24:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 898, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:56.845 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T12:24:57.624 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-16T12:24:57.624 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-16T12:24:57.624 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-16T12:24:57.624 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=11 2024-03-16T12:24:57.624 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T12:24:57.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T12:24:57.895 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:24:58.285 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:58 smithi153 ceph-mon[36901]: pgmap v508: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:58.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:58 smithi173 ceph-mon[36632]: pgmap v508: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:58.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:58 smithi098 ceph-mon[31633]: pgmap v508: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:24:59.394 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:24:59 smithi153 ceph-mon[36901]: pgmap v509: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:24:59.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:24:59 smithi098 ceph-mon[31633]: pgmap v509: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:24:59.568 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:24:59 smithi173 ceph-mon[36632]: pgmap v509: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T12:24:59.709 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 0, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 0, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 0, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.000968381, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000131787, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:24:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 121, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 116, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 450384, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 70096, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 2, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.015271700, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.004431021, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.011405154, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.002012749, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 116, 2024-03-16T12:24:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004711186, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000040613 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 450384, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 70096 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 1, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 87, 2024-03-16T12:24:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 160, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 93024, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.003524937, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.002365776, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000517685, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 85, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.003656662, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000043019 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:24:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 93024 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 0, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-03-16T12:24:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:24:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-03-16T12:24:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 148801, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 27, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-03-16T12:24:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 928, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:24:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 118, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 430873, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 118, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 118, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 430873, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-03-16T12:24:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-16T12:24:59.721 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-16T12:25:00.427 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-16T12:25:00.427 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-16T12:25:00.427 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-16T12:25:00.427 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-16T12:25:00.427 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T12:25:00.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T12:25:00.690 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:25:02.155 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:02 smithi173 ceph-mon[36632]: pgmap v510: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:25:02.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:02 smithi098 ceph-mon[31633]: pgmap v510: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:25:02.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:02 smithi153 ceph-mon[36901]: pgmap v510: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 789, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 791, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 843801, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 250912, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.045338826, 2024-03-16T12:25:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.013122191, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.012634768, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.014895492, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 789, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.017266953, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000021884 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 10, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 92, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 5000, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 296714, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-03-16T12:25:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.004709320, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.002418538, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001608474, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000078146, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 91, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.009233669, 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000101468 2024-03-16T12:25:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 295376 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 479, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 475, 2024-03-16T12:25:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 553663, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 182281, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 3, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.038452653, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.012391246, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.016110336, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.010482852, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 474, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.012215409, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000025770 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 466016, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 69072 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1912, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.033749088, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000017651 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 20, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-03-16T12:25:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 20, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 20, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-03-16T12:25:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-16T12:25:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:25:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-03-16T12:25:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:25:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-16T12:25:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 160, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-03-16T12:25:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4805760, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4803263, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 160 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 14, 2024-03-16T12:25:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 20, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 6, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 9, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 11, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 2, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 20, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 25, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 5, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-03-16T12:25:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 46328, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215420 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-16T12:25:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-16T12:25:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-16T12:25:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 3, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 2, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 1761, 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-16T12:25:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:25:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-16T12:25:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 189256, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 49, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-16T12:25:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8696, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 166, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 56858, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 272, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-16T12:25:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 1149, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 26, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 4, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 1149, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1149, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.394991200, 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000343769 2024-03-16T12:25:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1149, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1150 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 1149, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 1149, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 710, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 436, 2024-03-16T12:25:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 2, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:25:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-16T12:25:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 91, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 5, 2024-03-16T12:25:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 5, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-16T12:25:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1274, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1278450, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1274, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1274, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1278450, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1149, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 611466, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1149, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 611466, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1149, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 1149, 2024-03-16T12:25:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1149, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1149, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x557b10c566e0": { 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:02.617 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-16T12:25:03.349 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-16T12:25:03.349 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-16T12:25:03.349 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-16T12:25:03.349 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-16T12:25:03.349 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T12:25:03.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T12:25:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:25:04.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:04 smithi098 ceph-mon[31633]: pgmap v511: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:25:04.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:04 smithi153 ceph-mon[36901]: pgmap v511: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:25:04.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:04 smithi173 ceph-mon[36632]: pgmap v511: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-16T12:25:05.382 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-16T12:25:05.382 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:25:05.382 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 481, 2024-03-16T12:25:05.382 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 477, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 586416, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 183396, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.039395465, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.011616099, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.016655488, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.010215050, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 476, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.015132439, 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000031790 2024-03-16T12:25:05.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 466128, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 69552 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 4, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 89, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 876, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 295626, 2024-03-16T12:25:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.004352971, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.002565309, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001112662, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000054301, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 86, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006438853, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000074870 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 294928 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 568, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 568, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 149039, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 179807, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.030313109, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.009646185, 2024-03-16T12:25:05.386 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.008321748, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.010320362, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 567, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.013680591, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000024128 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:25:05.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1639, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.030992906, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000018909 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-16T12:25:05.388 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 0, 2024-03-16T12:25:05.390 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 1, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-16T12:25:05.391 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:25:05.392 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-16T12:25:05.393 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:25:05.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-16T12:25:05.395 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 3, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195202, 2024-03-16T12:25:05.396 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4194892, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 3 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 3, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 1, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 1, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-03-16T12:25:05.397 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 46448, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215420 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-03-16T12:25:05.398 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.400 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.401 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-16T12:25:05.402 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.405 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.406 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-16T12:25:05.407 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.408 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.409 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 1763, 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-16T12:25:05.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:25:05.411 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-16T12:25:05.412 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 181137, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 44, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:25:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8696, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 166, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 8464, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 13, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-16T12:25:05.414 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 924, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 3, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 924, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 924, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.421990624, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000456699 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.415 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 924, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 925 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 924, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 924, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 559, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 363, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-03-16T12:25:05.416 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:25:05.417 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-16T12:25:05.418 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 91, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 4, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 4, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-16T12:25:05.419 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.420 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1049, 2024-03-16T12:25:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 632561, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1049, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1049, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 632561, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 924, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 903, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 924, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 903, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.423 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 924, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 924, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 924, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 924, 2024-03-16T12:25:05.424 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:05.425 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55d8d87b86e0": { 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:05.426 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:05.428 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-16T12:25:06.022 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T12:25:06.022 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T12:25:06.022 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-16T12:25:06.022 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-16T12:25:06.022 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T12:25:06.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T12:25:06.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:06 smithi153 ceph-mon[36901]: pgmap v512: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:25:06.269 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:25:06.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:06 smithi173 ceph-mon[36632]: pgmap v512: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:25:06.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:06 smithi098 ceph-mon[31633]: pgmap v512: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-16T12:25:07.999 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-16T12:25:07.999 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 414, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 439, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 551945, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 275898, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 17, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.036668133, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.011654366, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.028724440, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.003355036, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 438, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.038392012, 2024-03-16T12:25:08.000 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000087652 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 515, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000067177, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000130 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 476448, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 72912 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 23, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 106, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 7960, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 271692, 2024-03-16T12:25:08.001 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 12, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.008178274, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.003142661, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004065428, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000277187, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 103, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.020781136, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000201758 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 11, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000001366, 2024-03-16T12:25:08.002 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000124 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 4592, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 219696 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 31, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 31, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 12786, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 20970, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 7, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.003616559, 2024-03-16T12:25:08.003 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000577767, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.002637133, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000359656, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 31, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.021408652, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000690601 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 31, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000011935, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000385 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:25:08.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 42, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001548053, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000036858 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-03-16T12:25:08.005 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000008242, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000004121 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 1, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 1, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000097060, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000097060 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.006 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 11, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 16, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 16, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 13, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 4, 2024-03-16T12:25:08.007 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 1, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 1, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 3, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 7, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 4, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 3, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 1000, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 1, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-03-16T12:25:08.008 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-03-16T12:25:08.009 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 11, 2024-03-16T12:25:08.010 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 7, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 8, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-03-16T12:25:08.011 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 3, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-03-16T12:25:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 3, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 2, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4195202, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-03-16T12:25:08.013 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003283294, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001641647 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 20, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 22, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 2, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 19, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 22, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 3, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 16, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 20, 2024-03-16T12:25:08.014 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 4, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 1, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 1, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 49020, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 223612 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 1, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 1, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 3, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.015 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-03-16T12:25:08.016 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-03-16T12:25:08.017 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000097060, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000097060 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.018 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-03-16T12:25:08.019 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.021 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.022 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.023 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-03-16T12:25:08.024 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-03-16T12:25:08.025 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.026 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 1, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 1, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 199, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-16T12:25:08.027 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:25:08.028 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:25:08.029 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 238407, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 78, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-16T12:25:08.030 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8696, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 166, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 80973, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 324, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-16T12:25:08.031 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 37, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 11110, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 28, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 37, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 37, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.239994695, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006486343 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.032 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 37, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 59 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 37, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 37, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 0, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 2, 2024-03-16T12:25:08.033 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 6, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 6, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-03-16T12:25:08.034 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-03-16T12:25:08.035 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 91, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 9, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 9, 2024-03-16T12:25:08.036 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 14, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 1 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-03-16T12:25:08.037 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 464, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 520195, 2024-03-16T12:25:08.038 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 464, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 464, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 520195, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:08.039 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 37, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 11110, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 37, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 11110, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 37, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 37, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 37, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 37, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.041 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:08.042 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x555d0b7c0500": { 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 3, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 898, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:08.043 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 3, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 898, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:08.044 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-16T12:25:08.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:08 smithi153 ceph-mon[36901]: pgmap v513: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:08.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:08 smithi173 ceph-mon[36632]: pgmap v513: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:08.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:08 smithi098 ceph-mon[31633]: pgmap v513: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:08.780 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-16T12:25:08.781 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-16T12:25:08.781 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-16T12:25:08.781 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=11 2024-03-16T12:25:08.781 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T12:25:08.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T12:25:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:25:10.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:10 smithi153 ceph-mon[36901]: pgmap v514: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:25:10.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:10 smithi173 ceph-mon[36632]: pgmap v514: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:25:10.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:10 smithi098 ceph-mon[31633]: pgmap v514: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T12:25:10.804 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 0, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 0, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.001112182, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000232363, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:25:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:25:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 121, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 118, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 448448, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 72208, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 2, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.016270837, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.004871834, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.011261355, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.002144609, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 117, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.004697895, 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000040152 2024-03-16T12:25:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 448448, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 72208 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 5, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 93, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 3184, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 172240, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-03-16T12:25:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.004464289, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.002594719, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.001205208, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 92, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006970056, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000075761 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-16T12:25:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 172240 2024-03-16T12:25:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-03-16T12:25:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:25:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:25:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 148801, 2024-03-16T12:25:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 27, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-03-16T12:25:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 928, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 122, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 431393, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-03-16T12:25:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 122, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 122, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 431393, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-03-16T12:25:10.816 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-16T12:25:11.500 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-16T12:25:11.501 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-16T12:25:11.501 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-16T12:25:11.501 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-16T12:25:11.501 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-16T12:25:11.512 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-16T12:25:11.512 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:25:11.512 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:25:11.575 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-16T12:25:11.575 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:25:11.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:25:11.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 -- 172.21.15.98:0/3155456898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4401055b0 msgr2=0x7fa44010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:11.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 --2- 172.21.15.98:0/3155456898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4401055b0 0x7fa44010be40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa438009ca0 tx=0x7fa43802c100 comp rx=0 tx=0).stop 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 -- 172.21.15.98:0/3155456898 shutdown_connections 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 --2- 172.21.15.98:0/3155456898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4401055b0 0x7fa44010be40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 --2- 172.21.15.98:0/3155456898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa440104bf0 0x7fa440105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 --2- 172.21.15.98:0/3155456898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440103a00 0x7fa440103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 -- 172.21.15.98:0/3155456898 >> 172.21.15.98:0/3155456898 conn(0x7fa4400fef90 msgr2=0x7fa4401013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 -- 172.21.15.98:0/3155456898 shutdown_connections 2024-03-16T12:25:11.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.856+0000 7fa448400640 1 -- 172.21.15.98:0/3155456898 wait complete. 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.857+0000 7fa448400640 1 Processor -- start 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 -- start start 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 0x7fa44019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4401055b0 0x7fa4401a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010f680 con 0x7fa4401055b0 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010f500 con 0x7fa440103a00 2024-03-16T12:25:11.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa448400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010f800 con 0x7fa440104bf0 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 0x7fa44019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33786/0 (socket says 172.21.15.98:33786) 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 -- 172.21.15.98:0/2187354329 learned_addr learned my addr 172.21.15.98:0/2187354329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 0x7fa44019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38958/0 (socket says 172.21.15.98:38958) 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa446400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4401055b0 0x7fa4401a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 -- 172.21.15.98:0/2187354329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 msgr2=0x7fa44019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 0x7fa44019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.858+0000 7fa445a00640 1 -- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4401055b0 msgr2=0x7fa4401a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa445a00640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4401055b0 0x7fa4401a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa445a00640 1 -- 172.21.15.98:0/2187354329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4401a1890 con 0x7fa440103a00 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa445a00640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e9c0 tx=0x7fa43400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43400cd80 con 0x7fa440103a00 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4340035f0 con 0x7fa440103a00 2024-03-16T12:25:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434010690 con 0x7fa440103a00 2024-03-16T12:25:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4401a1b80 con 0x7fa440103a00 2024-03-16T12:25:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.859+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401a2080 con 0x7fa440103a00 2024-03-16T12:25:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.860+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa44010ad20 con 0x7fa440103a00 2024-03-16T12:25:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.862+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa4340037a0 con 0x7fa440103a00 2024-03-16T12:25:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.863+0000 7fa43e000640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 0x7fa4280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.863+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa4340cf180 con 0x7fa440103a00 2024-03-16T12:25:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.864+0000 7fa445000640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 0x7fa4280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.864+0000 7fa445000640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 0x7fa4280a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa42c005d00 tx=0x7fa42c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:11.867+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa43409be60 con 0x7fa440103a00 2024-03-16T12:25:12.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.072+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa44002d350 con 0x7fa440103a00 2024-03-16T12:25:12.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.076+0000 7fa43e000640 1 -- 172.21.15.98:0/2187354329 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7fa4340860e0 con 0x7fa440103a00 2024-03-16T12:25:12.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:25:12.056 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.078+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 msgr2=0x7fa4280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.078+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 0x7fa4280a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa42c005d00 tx=0x7fa42c00b410 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.078+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 msgr2=0x7fa440199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.078+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e9c0 tx=0x7fa43400ee80 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 shutdown_connections 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa4280a2880 0x7fa4280a4d40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4401055b0 0x7fa4401a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa440104bf0 0x7fa44019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 --2- 172.21.15.98:0/2187354329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440103a00 0x7fa440199c80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 >> 172.21.15.98:0/2187354329 conn(0x7fa4400fef90 msgr2=0x7fa440107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:12.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 shutdown_connections 2024-03-16T12:25:12.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.079+0000 7fa448400640 1 -- 172.21.15.98:0/2187354329 wait complete. 2024-03-16T12:25:12.078 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T12:13:27.314440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T12:18:38.814265+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T12:18:39.961579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T12:18:43.077289+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T12:25:12.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 -- 172.21.15.98:0/1863729823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 msgr2=0x7ff808108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1863729823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff808108120 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009900 tx=0x7ff7fc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 -- 172.21.15.98:0/1863729823 shutdown_connections 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1863729823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8081098b0 0x7ff808110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1863729823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff808108ef0 0x7ff808109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1863729823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff808108120 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 -- 172.21.15.98:0/1863729823 >> 172.21.15.98:0/1863729823 conn(0x7ff808076cb0 msgr2=0x7ff8080790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.307+0000 7ff80ec00640 1 -- 172.21.15.98:0/1863729823 shutdown_connections 2024-03-16T12:25:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.308+0000 7ff80ec00640 1 -- 172.21.15.98:0/1863729823 wait complete. 2024-03-16T12:25:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.308+0000 7ff80ec00640 1 Processor -- start 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 -- start start 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff808108ef0 0x7ff80819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 0x7ff8081a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8081139a0 con 0x7ff808108ef0 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff808113820 con 0x7ff8081098b0 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff808113b20 con 0x7ff808107d00 2024-03-16T12:25:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:38976/0 (socket says 172.21.15.98:38976) 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80d800640 1 -- 172.21.15.98:0/1747907253 learned_addr learned my addr 172.21.15.98:0/1747907253 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80e200640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 0x7ff8081a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.309+0000 7ff80ce00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff808108ef0 0x7ff80819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 -- 172.21.15.98:0/1747907253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 msgr2=0x7ff8081a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 0x7ff8081a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 -- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff808108ef0 msgr2=0x7ff80819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff808108ef0 0x7ff80819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 -- 172.21.15.98:0/1747907253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8081a5bf0 con 0x7ff808107d00 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80e200640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 0x7ff8081a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80d800640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc000c00 tx=0x7ff7fc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc043020 con 0x7ff808107d00 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc0054e0 con 0x7ff808107d00 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc003c40 con 0x7ff808107d00 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8081a5e80 con 0x7ff808107d00 2024-03-16T12:25:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.310+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8081a6290 con 0x7ff808107d00 2024-03-16T12:25:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.311+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff808068980 con 0x7ff808107d00 2024-03-16T12:25:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.313+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff7fc046020 con 0x7ff808107d00 2024-03-16T12:25:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.314+0000 7ff806000640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 0x7ff7f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.315+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7fc0f0db0 con 0x7ff808107d00 2024-03-16T12:25:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.315+0000 7ff80ce00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 0x7ff7f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.315+0000 7ff80ce00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 0x7ff7f00a4d40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff80819f4e0 tx=0x7ff7f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.318+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7fc0f5070 con 0x7ff808107d00 2024-03-16T12:25:12.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:12 smithi173 ceph-mon[36632]: pgmap v515: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:25:12.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:12 smithi098 ceph-mon[31633]: pgmap v515: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:25:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.516+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8080623e0 con 0x7ff808107d00 2024-03-16T12:25:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.516+0000 7ff806000640 1 -- 172.21.15.98:0/1747907253 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ff7fc04d020 con 0x7ff808107d00 2024-03-16T12:25:12.496 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-16T12:25:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.519+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 msgr2=0x7ff7f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.519+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 0x7ff7f00a4d40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff80819f4e0 tx=0x7ff7f4005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.519+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 msgr2=0x7ff80819e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.519+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc000c00 tx=0x7ff7fc02c650 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 shutdown_connections 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7ff7f00a2880 0x7ff7f00a4d40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8081098b0 0x7ff8081a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff808108ef0 0x7ff80819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 --2- 172.21.15.98:0/1747907253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff808107d00 0x7ff80819e070 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 >> 172.21.15.98:0/1747907253 conn(0x7ff808076cb0 msgr2=0x7ff80810c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 shutdown_connections 2024-03-16T12:25:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.520+0000 7ff80ec00640 1 -- 172.21.15.98:0/1747907253 wait complete. 2024-03-16T12:25:12.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:12 smithi153 ceph-mon[36901]: pgmap v515: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T12:25:12.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:25:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 -- 172.21.15.98:0/1148603692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 msgr2=0x7fa328101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 --2- 172.21.15.98:0/1148603692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328101c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa318009ca0 tx=0x7fa31802c100 comp rx=0 tx=0).stop 2024-03-16T12:25:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 -- 172.21.15.98:0/1148603692 shutdown_connections 2024-03-16T12:25:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 --2- 172.21.15.98:0/1148603692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281033a0 0x7fa328109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 --2- 172.21.15.98:0/1148603692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281029e0 0x7fa328102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 --2- 172.21.15.98:0/1148603692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328101c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 -- 172.21.15.98:0/1148603692 >> 172.21.15.98:0/1148603692 conn(0x7fa3280fcd80 msgr2=0x7fa3280ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 -- 172.21.15.98:0/1148603692 shutdown_connections 2024-03-16T12:25:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.747+0000 7fa32ea00640 1 -- 172.21.15.98:0/1148603692 wait complete. 2024-03-16T12:25:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.748+0000 7fa32ea00640 1 Processor -- start 2024-03-16T12:25:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 -- start start 2024-03-16T12:25:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281033a0 0x7fa3281a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa32810d440 con 0x7fa3281017f0 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa32810d2c0 con 0x7fa3281033a0 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa32810d5c0 con 0x7fa3281029e0 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa327e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa327400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa327400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39006/0 (socket says 172.21.15.98:39006) 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa327400640 1 -- 172.21.15.98:0/2554625507 learned_addr learned my addr 172.21.15.98:0/2554625507 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.749+0000 7fa32ca00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281033a0 0x7fa3281a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 -- 172.21.15.98:0/2554625507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281033a0 msgr2=0x7fa3281a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281033a0 0x7fa3281a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 -- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 msgr2=0x7fa328199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 -- 172.21.15.98:0/2554625507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3281a1970 con 0x7fa3281029e0 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327e00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa327400640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa31000ea60 tx=0x7fa31000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31000cde0 con 0x7fa3281029e0 2024-03-16T12:25:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3100035f0 con 0x7fa3281029e0 2024-03-16T12:25:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3281a1c60 con 0x7fa3281029e0 2024-03-16T12:25:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310010670 con 0x7fa3281029e0 2024-03-16T12:25:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.750+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3281a21b0 con 0x7fa3281029e0 2024-03-16T12:25:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.751+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3281a23e0 con 0x7fa3281029e0 2024-03-16T12:25:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.754+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa3100037a0 con 0x7fa3281029e0 2024-03-16T12:25:12.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.755+0000 7fa324c00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 0x7fa30c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:12.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.755+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa3100cf100 con 0x7fa3281029e0 2024-03-16T12:25:12.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.755+0000 7fa327e00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 0x7fa30c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:12.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.756+0000 7fa327e00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 0x7fa30c0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa31802c640 tx=0x7fa318009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.759+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa31009bde0 con 0x7fa3281029e0 2024-03-16T12:25:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.958+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3281a2650 con 0x7fa3281029e0 2024-03-16T12:25:12.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.959+0000 7fa324c00640 1 -- 172.21.15.98:0/2554625507 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fa3100860f0 con 0x7fa3281029e0 2024-03-16T12:25:12.939 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.961+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 msgr2=0x7fa30c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.961+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 0x7fa30c0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa31802c640 tx=0x7fa318009360 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.961+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 msgr2=0x7fa32819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.961+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa31000ea60 tx=0x7fa31000ef20 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 shutdown_connections 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fa30c0a2800 0x7fa30c0a4cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3281033a0 0x7fa3281a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3281029e0 0x7fa32819a290 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 --2- 172.21.15.98:0/2554625507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3281017f0 0x7fa328199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 >> 172.21.15.98:0/2554625507 conn(0x7fa3280fcd80 msgr2=0x7fa328108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 shutdown_connections 2024-03-16T12:25:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:12.962+0000 7fa32ea00640 1 -- 172.21.15.98:0/2554625507 wait complete. 2024-03-16T12:25:12.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 -- 172.21.15.98:0/3421360614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 msgr2=0x7f0504105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 --2- 172.21.15.98:0/3421360614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f0504105090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009900 tx=0x7f04f002bc30 comp rx=0 tx=0).stop 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 -- 172.21.15.98:0/3421360614 shutdown_connections 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 --2- 172.21.15.98:0/3421360614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 0x7f050410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 --2- 172.21.15.98:0/3421360614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f0504105090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 --2- 172.21.15.98:0/3421360614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.198+0000 7f050a400640 1 -- 172.21.15.98:0/3421360614 >> 172.21.15.98:0/3421360614 conn(0x7f05040fef90 msgr2=0x7f05041013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.199+0000 7f050a400640 1 -- 172.21.15.98:0/3421360614 shutdown_connections 2024-03-16T12:25:13.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.199+0000 7f050a400640 1 -- 172.21.15.98:0/3421360614 wait complete. 2024-03-16T12:25:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 Processor -- start 2024-03-16T12:25:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 -- start start 2024-03-16T12:25:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f050419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 0x7f05041a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f6f0 con 0x7f05041055d0 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f570 con 0x7f0504103a20 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f050a400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f870 con 0x7f0504104c10 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f0503400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f0502a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f050419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f0503e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 0x7f05041a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f0503400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33826/0 (socket says 172.21.15.98:33826) 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.200+0000 7f0503400640 1 -- 172.21.15.98:0/1479367460 learned_addr learned my addr 172.21.15.98:0/1479367460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 -- 172.21.15.98:0/1479367460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 msgr2=0x7f0504199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 -- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 msgr2=0x7f05041a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 0x7f05041a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 -- 172.21.15.98:0/1479367460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05041a1960 con 0x7f0504104c10 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0503400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.201+0000 7f0502a00640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f050419a290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f04f0036040 tx=0x7f04f002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.202+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0043020 con 0x7f0504104c10 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.202+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f00054e0 con 0x7f0504104c10 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.202+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0003c10 con 0x7f0504104c10 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.202+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05041a1bf0 con 0x7f0504104c10 2024-03-16T12:25:13.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.202+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05041a20c0 con 0x7f0504104c10 2024-03-16T12:25:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.203+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f050410ad40 con 0x7f0504104c10 2024-03-16T12:25:13.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.204+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f04f0046020 con 0x7f0504104c10 2024-03-16T12:25:13.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.206+0000 7f04fbe00640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 0x7f04e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.206+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f04f00f0d50 con 0x7f0504104c10 2024-03-16T12:25:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.206+0000 7f0503400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 0x7f04e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.207+0000 7f0503400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 0x7f04e80a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f050419b220 tx=0x7f04f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.210+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04f00f4030 con 0x7f0504104c10 2024-03-16T12:25:13.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:13 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2187354329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:13.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:13 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1747907253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:13 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2554625507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:13 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2187354329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:13.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:13 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1747907253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:13 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2554625507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.411+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0504062480 con 0x7f0504104c10 2024-03-16T12:25:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.412+0000 7f04fbe00640 1 -- 172.21.15.98:0/1479367460 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f04f004d020 con 0x7f0504104c10 2024-03-16T12:25:13.392 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.414+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 msgr2=0x7f04e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.414+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 0x7f04e80a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f050419b220 tx=0x7f04f4005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.414+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 msgr2=0x7f050419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.414+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f050419a290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f04f0036040 tx=0x7f04f002c650 comp rx=0 tx=0).stop 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 shutdown_connections 2024-03-16T12:25:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f04e80a2880 0x7f04e80a4d40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05041055d0 0x7f05041a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0504104c10 0x7f050419a290 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 --2- 172.21.15.98:0/1479367460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0504103a20 0x7f0504199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 >> 172.21.15.98:0/1479367460 conn(0x7f05040fef90 msgr2=0x7f0504107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 shutdown_connections 2024-03-16T12:25:13.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.415+0000 7f050a400640 1 -- 172.21.15.98:0/1479367460 wait complete. 2024-03-16T12:25:13.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T12:25:13.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:13 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2187354329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:13.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:13 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1747907253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:13 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2554625507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 -- 172.21.15.98:0/3747231675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40773e0 msgr2=0x7f79d4077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 --2- 172.21.15.98:0/3747231675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40773e0 0x7f79d4077840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009860 tx=0x7f79c002bb70 comp rx=0 tx=0).stop 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 -- 172.21.15.98:0/3747231675 shutdown_connections 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 --2- 172.21.15.98:0/3747231675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d4077d80 0x7f79d4078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 --2- 172.21.15.98:0/3747231675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40773e0 0x7f79d4077840 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 --2- 172.21.15.98:0/3747231675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d40761a0 0x7f79d40765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 -- 172.21.15.98:0/3747231675 >> 172.21.15.98:0/3747231675 conn(0x7f79d40fef90 msgr2=0x7f79d41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 -- 172.21.15.98:0/3747231675 shutdown_connections 2024-03-16T12:25:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.650+0000 7f79da400640 1 -- 172.21.15.98:0/3747231675 wait complete. 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.651+0000 7f79da400640 1 Processor -- start 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.651+0000 7f79da400640 1 -- start start 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 0x7f79d419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d4077d80 0x7f79d41a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113ab0 con 0x7f79d4077d80 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113930 con 0x7f79d40761a0 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79da400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4113c30 con 0x7f79d40773e0 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d3400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 0x7f79d419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d3400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33838/0 (socket says 172.21.15.98:33838) 2024-03-16T12:25:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d3400640 1 -- 172.21.15.98:0/3996452197 learned_addr learned my addr 172.21.15.98:0/3996452197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d3e00640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d4077d80 0x7f79d41a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.652+0000 7f79d2a00640 1 -- 172.21.15.98:0/3996452197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 msgr2=0x7f79d419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d2a00640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d2a00640 1 -- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d4077d80 msgr2=0x7f79d41a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d2a00640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d4077d80 0x7f79d41a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d2a00640 1 -- 172.21.15.98:0/3996452197 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d41a5d00 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d3400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79d2a00640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 0x7f79d419e620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f79c002c0b0 tx=0x7f79c0032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0041020 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c002c9f0 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0039040 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d41a5f90 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.653+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d41a63a0 con 0x7f79d40773e0 2024-03-16T12:25:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.655+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d4110530 con 0x7f79d40773e0 2024-03-16T12:25:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.656+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f79c0044020 con 0x7f79d40773e0 2024-03-16T12:25:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.658+0000 7f79cbe00640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 0x7f79b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.658+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f79c00f05b0 con 0x7f79d40773e0 2024-03-16T12:25:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.658+0000 7f79d3400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 0x7f79b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.659+0000 7f79d3400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 0x7f79b80a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f79c4011550 tx=0x7f79c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.662+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79c004b020 con 0x7f79d40773e0 2024-03-16T12:25:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.862+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f79d40624b0 con 0x7f79d40773e0 2024-03-16T12:25:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.863+0000 7f79cbe00640 1 -- 172.21.15.98:0/3996452197 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f79c00bd2f0 con 0x7f79d40773e0 2024-03-16T12:25:13.843 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 msgr2=0x7f79b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 0x7f79b80a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f79c4011550 tx=0x7f79c4005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 msgr2=0x7f79d419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 0x7f79d419e620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f79c002c0b0 tx=0x7f79c0032430 comp rx=0 tx=0).stop 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 shutdown_connections 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f79b80a2800 0x7f79b80a4cc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.865+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d4077d80 0x7f79d41a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.866+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79d40773e0 0x7f79d419e620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.866+0000 7f79da400640 1 --2- 172.21.15.98:0/3996452197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79d40761a0 0x7f79d419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.866+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 >> 172.21.15.98:0/3996452197 conn(0x7f79d40fef90 msgr2=0x7f79d4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.866+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 shutdown_connections 2024-03-16T12:25:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:13.866+0000 7f79da400640 1 -- 172.21.15.98:0/3996452197 wait complete. 2024-03-16T12:25:13.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T12:25:14.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 -- 172.21.15.98:0/4149098652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 msgr2=0x7fb7e8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 --2- 172.21.15.98:0/4149098652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e8109470 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009900 tx=0x7fb7d402bc30 comp rx=0 tx=0).stop 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 -- 172.21.15.98:0/4149098652 shutdown_connections 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 --2- 172.21.15.98:0/4149098652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 0x7fb7e8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 --2- 172.21.15.98:0/4149098652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e8109470 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 --2- 172.21.15.98:0/4149098652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 -- 172.21.15.98:0/4149098652 >> 172.21.15.98:0/4149098652 conn(0x7fb7e8076da0 msgr2=0x7fb7e80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 -- 172.21.15.98:0/4149098652 shutdown_connections 2024-03-16T12:25:14.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.095+0000 7fb7f0400640 1 -- 172.21.15.98:0/4149098652 wait complete. 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.096+0000 7fb7f0400640 1 Processor -- start 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.096+0000 7fb7f0400640 1 -- start start 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 0x7fb7e81a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e8113ad0 con 0x7fb7e8107e00 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e8113950 con 0x7fb7e8108ff0 2024-03-16T12:25:14.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7f0400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e8113c50 con 0x7fb7e81099b0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7eda00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7ed000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7ee400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 0x7fb7e81a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7eda00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e819e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57862/0 (socket says 172.21.15.98:57862) 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7eda00640 1 -- 172.21.15.98:0/3690655671 learned_addr learned my addr 172.21.15.98:0/3690655671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7ed000640 1 -- 172.21.15.98:0/3690655671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 msgr2=0x7fb7e81a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.097+0000 7fb7ed000640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 0x7fb7e81a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7ed000640 1 -- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 msgr2=0x7fb7e819e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7ed000640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e819e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7ed000640 1 -- 172.21.15.98:0/3690655671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e81a5d00 con 0x7fb7e8108ff0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7ed000640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e819e620 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4036040 tx=0x7fb7d402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4043020 con 0x7fb7e8108ff0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7d40054e0 con 0x7fb7e8108ff0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e81a5f90 con 0x7fb7e8108ff0 2024-03-16T12:25:14.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4003ca0 con 0x7fb7e8108ff0 2024-03-16T12:25:14.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.098+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e81a6450 con 0x7fb7e8108ff0 2024-03-16T12:25:14.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.099+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e810f0c0 con 0x7fb7e8108ff0 2024-03-16T12:25:14.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.101+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb7d4046020 con 0x7fb7e8108ff0 2024-03-16T12:25:14.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.102+0000 7fb7e6000640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.102+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb7d40f0e70 con 0x7fb7e8108ff0 2024-03-16T12:25:14.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.103+0000 7fb7eda00640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.103+0000 7fb7eda00640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00bc80 tx=0x7fb7dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:14.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.111+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7d404d020 con 0x7fb7e8108ff0 2024-03-16T12:25:14.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:14 smithi098 ceph-mon[31633]: pgmap v516: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:14.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:14 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1479367460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:14 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3996452197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.323+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb7e8062410 con 0x7fb7e8108ff0 2024-03-16T12:25:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.323+0000 7fb7e6000640 1 -- 172.21.15.98:0/3690655671 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+588 (secure 0 0 0) 0x7fb7d40bdb50 con 0x7fb7e8108ff0 2024-03-16T12:25:14.303 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3396936170 2024-03-17T12:10:42.632912+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/352508062 2024-03-17T12:10:42.632912+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/653650228 2024-03-17T12:10:08.286891+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/719012091 2024-03-17T12:10:42.632912+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/196846375 2024-03-17T12:10:42.632912+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3496387635 2024-03-17T12:10:08.286891+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3016831032 2024-03-17T12:10:08.286891+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3029136477 2024-03-17T12:10:08.286891+0000 2024-03-16T12:25:14.304 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-03-16T12:25:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 msgr2=0x7fb7d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00bc80 tx=0x7fb7dc00b410 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 msgr2=0x7fb7e819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e819e620 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4036040 tx=0x7fb7d402c6c0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 shutdown_connections 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7e81099b0 0x7fb7e81a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e8108ff0 0x7fb7e819e620 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 --2- 172.21.15.98:0/3690655671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7e8107e00 0x7fb7e819e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 >> 172.21.15.98:0/3690655671 conn(0x7fb7e8076da0 msgr2=0x7fb7e810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 shutdown_connections 2024-03-16T12:25:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.326+0000 7fb7f0400640 1 -- 172.21.15.98:0/3690655671 wait complete. 2024-03-16T12:25:14.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:14 smithi173 ceph-mon[36632]: pgmap v516: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:14.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:14 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1479367460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:14 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3996452197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.327 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-16T12:25:14.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:25:14.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-16T12:25:14.444 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:25:14.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:25:14.444 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T12:25:14.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:25:14.501 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-03-16T12:25:14.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:14 smithi153 ceph-mon[36901]: pgmap v516: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T12:25:14.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:14 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1479367460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:14 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3996452197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T12:25:14.540 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T12:25:14.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:25:14.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T12:25:14.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:25:14.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-03-16T12:25:14.779 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T12:25:14.789 INFO:tasks.cephadm:Teardown begin 2024-03-16T12:25:14.789 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T12:25:14.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:25:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 -- 172.21.15.98:0/2886590268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 msgr2=0x7f6a54105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 --2- 172.21.15.98:0/2886590268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a54105070 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:25:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 -- 172.21.15.98:0/2886590268 shutdown_connections 2024-03-16T12:25:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 --2- 172.21.15.98:0/2886590268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a541055b0 0x7f6a5410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 --2- 172.21.15.98:0/2886590268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a54105070 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 --2- 172.21.15.98:0/2886590268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a54103a00 0x7f6a54103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 -- 172.21.15.98:0/2886590268 >> 172.21.15.98:0/2886590268 conn(0x7f6a540fef90 msgr2=0x7f6a541013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 -- 172.21.15.98:0/2886590268 shutdown_connections 2024-03-16T12:25:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.997+0000 7f6a5a000640 1 -- 172.21.15.98:0/2886590268 wait complete. 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.998+0000 7f6a5a000640 1 Processor -- start 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.998+0000 7f6a5a000640 1 -- start start 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 0x7f6a54199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a5419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5410f6b0 con 0x7f6a54103a00 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5410f530 con 0x7f6a541055b0 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a5a000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5410f830 con 0x7f6a54104bf0 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 0x7f6a54199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 0x7f6a54199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57890/0 (socket says 172.21.15.98:57890) 2024-03-16T12:25:14.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53400640 1 -- 172.21.15.98:0/4168553726 learned_addr learned my addr 172.21.15.98:0/4168553726 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33862/0 (socket says 172.21.15.98:33862) 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a4aa00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a5419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 -- 172.21.15.98:0/4168553726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 msgr2=0x7f6a5419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a5419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 -- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 msgr2=0x7f6a54199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 0x7f6a54199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:14.999+0000 7f6a53e00640 1 -- 172.21.15.98:0/4168553726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a541a1920 con 0x7f6a541055b0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a53e00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00d680 tx=0x7f6a4c00db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c017070 con 0x7f6a541055b0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a541a1c10 con 0x7f6a541055b0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a4c009820 con 0x7f6a541055b0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c004e00 con 0x7f6a541055b0 2024-03-16T12:25:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.000+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a541a2110 con 0x7f6a541055b0 2024-03-16T12:25:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.001+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a5410ad20 con 0x7f6a541055b0 2024-03-16T12:25:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.003+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a4c0099d0 con 0x7f6a541055b0 2024-03-16T12:25:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.004+0000 7f6a50c00640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 0x7f6a380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.004+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6a4c0cea70 con 0x7f6a541055b0 2024-03-16T12:25:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.005+0000 7f6a53400640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 0x7f6a380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.005+0000 7f6a53400640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 0x7f6a380a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6a5419b1e0 tx=0x7f6a44009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.008+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a4c09b100 con 0x7f6a541055b0 2024-03-16T12:25:15.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.252+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a540623e0 con 0x7f6a541055b0 2024-03-16T12:25:15.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.254+0000 7f6a50c00640 1 -- 172.21.15.98:0/4168553726 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7f6a4c09f1a0 con 0x7f6a541055b0 2024-03-16T12:25:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:25:15.234 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:25:15.234 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:25:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 msgr2=0x7f6a380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 0x7f6a380a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6a5419b1e0 tx=0x7f6a44009500 comp rx=0 tx=0).stop 2024-03-16T12:25:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 msgr2=0x7f6a541a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00d680 tx=0x7f6a4c00db40 comp rx=0 tx=0).stop 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 shutdown_connections 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f6a380a2800 0x7f6a380a4cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a541055b0 0x7f6a541a1280 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a54104bf0 0x7f6a5419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 --2- 172.21.15.98:0/4168553726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a54103a00 0x7f6a54199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 >> 172.21.15.98:0/4168553726 conn(0x7f6a540fef90 msgr2=0x7f6a54107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.257+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 shutdown_connections 2024-03-16T12:25:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.258+0000 7f6a5a000640 1 -- 172.21.15.98:0/4168553726 wait complete. 2024-03-16T12:25:15.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:15 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/3690655671' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:25:15.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:25:15.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:15 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/3690655671' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:25:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 -- 172.21.15.98:0/959837311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89501055b0 msgr2=0x7f895010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 --2- 172.21.15.98:0/959837311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89501055b0 0x7f895010be40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f894c009ca0 tx=0x7f894c02c100 comp rx=0 tx=0).stop 2024-03-16T12:25:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 -- 172.21.15.98:0/959837311 shutdown_connections 2024-03-16T12:25:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 --2- 172.21.15.98:0/959837311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89501055b0 0x7f895010be40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 --2- 172.21.15.98:0/959837311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 0x7f8950105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 --2- 172.21.15.98:0/959837311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8950103a00 0x7f8950103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 -- 172.21.15.98:0/959837311 >> 172.21.15.98:0/959837311 conn(0x7f89500fef90 msgr2=0x7f89501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:15.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 -- 172.21.15.98:0/959837311 shutdown_connections 2024-03-16T12:25:15.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.489+0000 7f8958800640 1 -- 172.21.15.98:0/959837311 wait complete. 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.490+0000 7f8958800640 1 Processor -- start 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.490+0000 7f8958800640 1 -- start start 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 0x7f8950199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 0x7f895019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f895010f660 con 0x7f8950104bf0 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f895010f4e0 con 0x7f8950103a00 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8958800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f895010f7e0 con 0x7f89501055b0 2024-03-16T12:25:15.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8955400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 0x7f895019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8955e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 0x7f8950199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8955e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 0x7f8950199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:33888/0 (socket says 172.21.15.98:33888) 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39090/0 (socket says 172.21.15.98:39090) 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8955e00640 1 -- 172.21.15.98:0/165176047 learned_addr learned my addr 172.21.15.98:0/165176047 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 -- 172.21.15.98:0/165176047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 msgr2=0x7f8950199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 0x7f8950199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 -- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 msgr2=0x7f895019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 0x7f895019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.491+0000 7f8956800640 1 -- 172.21.15.98:0/165176047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89501a18b0 con 0x7f89501055b0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f8956800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f894c02c640 tx=0x7f894c035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c039040 con 0x7f89501055b0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894c044070 con 0x7f89501055b0 2024-03-16T12:25:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c0043e0 con 0x7f89501055b0 2024-03-16T12:25:15.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89501a1b40 con 0x7f89501055b0 2024-03-16T12:25:15.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.492+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89501a1fb0 con 0x7f89501055b0 2024-03-16T12:25:15.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.493+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895010acc0 con 0x7f89501055b0 2024-03-16T12:25:15.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.495+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f894c004580 con 0x7f89501055b0 2024-03-16T12:25:15.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.496+0000 7f894aa00640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 0x7f89380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.496+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f894c0f13c0 con 0x7f89501055b0 2024-03-16T12:25:15.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.496+0000 7f8955e00640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 0x7f89380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.497+0000 7f8955e00640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 0x7f89380a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f895019ab40 tx=0x7f8944005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:15.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.500+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f894c0be100 con 0x7f89501055b0 2024-03-16T12:25:15.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:15 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/3690655671' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:25:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.736+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f895002d1b0 con 0x7f89501055b0 2024-03-16T12:25:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.737+0000 7f894aa00640 1 -- 172.21.15.98:0/165176047 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11873 (secure 0 0 0) 0x7f894c0bda00 con 0x7f89501055b0 2024-03-16T12:25:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:25:15.718 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/211337722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":211337722},{"type":"v1","addr":"172.21.15.98:6835","nonce":211337722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/1905625242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":1905625242},{"type":"v1","addr":"172.21.15.98:6839","nonce":1905625242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1343544864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1343544864},{"type":"v1","addr":"172.21.15.98:6841","nonce":1343544864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/307291521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":307291521},{"type":"v1","addr":"172.21.15.173:6833","nonce":307291521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3502700465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3502700465},{"type":"v1","addr":"172.21.15.153:6833","nonce":3502700465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3487564371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3487564371},{"type":"v1","addr":"172.21.15.98:6837","nonce":3487564371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T12:18:40.454968+0000","modified":"2024-03-16T12:21:33.620740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24599,"mds_1":24553,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6835/4013234440","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4013234440},{"type":"v1","addr":"172.21.15.153:6835","nonce":4013234440}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2799668647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2799668647},{"type":"v1","addr":"172.21.15.173:6835","nonce":2799668647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/4083501894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":4083501894},{"type":"v1","addr":"172.21.15.173:6837","nonce":4083501894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.153:6837/419588181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":419588181},{"type":"v1","addr":"172.21.15.153:6837","nonce":419588181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6839/205594788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":205594788},{"type":"v1","addr":"172.21.15.173:6839","nonce":205594788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/542416413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":542416413},{"type":"v1","addr":"172.21.15.153:6839","nonce":542416413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24542,24553,24599]},"id":1}]} 2024-03-16T12:25:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-03-16T12:25:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.739+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 msgr2=0x7f89380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.739+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 0x7f89380a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f895019ab40 tx=0x7f8944005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.739+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 msgr2=0x7f89501a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.739+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f894c02c640 tx=0x7f894c035d90 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 shutdown_connections 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f89380a2800 0x7f89380a4cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89501055b0 0x7f89501a1210 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8950104bf0 0x7f895019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 --2- 172.21.15.98:0/165176047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8950103a00 0x7f8950199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 >> 172.21.15.98:0/165176047 conn(0x7f89500fef90 msgr2=0x7f8950107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 shutdown_connections 2024-03-16T12:25:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.740+0000 7f8958800640 1 -- 172.21.15.98:0/165176047 wait complete. 2024-03-16T12:25:15.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:25:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 -- 172.21.15.98:0/160762499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7034105390 msgr2=0x7f703410bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 --2- 172.21.15.98:0/160762499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7034105390 0x7f703410bc20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f702c009ca0 tx=0x7f702c02c100 comp rx=0 tx=0).stop 2024-03-16T12:25:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 -- 172.21.15.98:0/160762499 shutdown_connections 2024-03-16T12:25:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 --2- 172.21.15.98:0/160762499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7034105390 0x7f703410bc20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 --2- 172.21.15.98:0/160762499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 0x7f7034104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 --2- 172.21.15.98:0/160762499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70341037e0 0x7f7034103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 -- 172.21.15.98:0/160762499 >> 172.21.15.98:0/160762499 conn(0x7f70340fed50 msgr2=0x7f7034101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:15.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 -- 172.21.15.98:0/160762499 shutdown_connections 2024-03-16T12:25:15.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.967+0000 7f7038e00640 1 -- 172.21.15.98:0/160762499 wait complete. 2024-03-16T12:25:15.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.968+0000 7f7038e00640 1 Processor -- start 2024-03-16T12:25:15.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.968+0000 7f7038e00640 1 -- start start 2024-03-16T12:25:15.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.968+0000 7f7038e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7038e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 0x7f7034199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7038e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 0x7f70341a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7038e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f703410f460 con 0x7f7034105390 2024-03-16T12:25:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7038e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f703410f2e0 con 0x7f70341049d0 2024-03-16T12:25:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7038e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f703410f5e0 con 0x7f70341037e0 2024-03-16T12:25:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39098/0 (socket says 172.21.15.98:39098) 2024-03-16T12:25:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 -- 172.21.15.98:0/66366954 learned_addr learned my addr 172.21.15.98:0/66366954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7032a00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 0x7f7034199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 0x7f70341a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 -- 172.21.15.98:0/66366954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 msgr2=0x7f7034199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 0x7f7034199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 -- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 msgr2=0x7f70341a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 0x7f70341a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 -- 172.21.15.98:0/66366954 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70341a16a0 con 0x7f70341037e0 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 0x7f70341a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.969+0000 7f7033400640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f702400d950 tx=0x7f702400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.970+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024017020 con 0x7f70341037e0 2024-03-16T12:25:15.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.970+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7024009c30 con 0x7f70341037e0 2024-03-16T12:25:15.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.970+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702400f720 con 0x7f70341037e0 2024-03-16T12:25:15.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.970+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70341a1990 con 0x7f70341037e0 2024-03-16T12:25:15.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.970+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70341a1e90 con 0x7f70341037e0 2024-03-16T12:25:15.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.971+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703410ab00 con 0x7f70341037e0 2024-03-16T12:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.973+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f702401a020 con 0x7f70341037e0 2024-03-16T12:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.974+0000 7f702be00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 0x7f70180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.974+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f70240ced10 con 0x7f70341037e0 2024-03-16T12:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.974+0000 7f7032a00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 0x7f70180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.975+0000 7f7032a00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 0x7f70180a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f703419af90 tx=0x7f7020005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:15.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:15.978+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f70240d4020 con 0x7f70341037e0 2024-03-16T12:25:16.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.178+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f703402d460 con 0x7f70341037e0 2024-03-16T12:25:16.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.181+0000 7f702be00640 1 -- 172.21.15.98:0/66366954 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24654 (secure 0 0 0) 0x7f7024021020 con 0x7f70341037e0 2024-03-16T12:25:16.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-16T12:25:16.161 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"e9cfddf2-e38d-11ee-95c9-87774f69a715","created":"2024-03-16T12:09:34.983809+0000","modified":"2024-03-16T12:19:44.538706+0000","last_up_change":"2024-03-16T12:17:43.354643+0000","last_in_change":"2024-03-16T12:17:22.672594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T12:13:27.314440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T12:18:38.814265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T12:18:39.961579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T12:18:43.077289+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"464e1266-b151-4bf9-82b5-398b1aa8d1db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6803","nonce":3304246268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6805","nonce":3304246268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6809","nonce":3304246268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3304246268},{"type":"v1","addr":"172.21.15.98:6807","nonce":3304246268}]},"public_addr":"172.21.15.98:6803/3304246268","cluster_addr":"172.21.15.98:6805/3304246268","heartbeat_back_addr":"172.21.15.98:6809/3304246268","heartbeat_front_addr":"172.21.15.98:6807/3304246268","state":["exists","up"]},{"osd":1,"uuid":"eb71b246-921e-49df-9e6d-08cd3431f968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6801","nonce":4082140415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6803","nonce":4082140415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6807","nonce":4082140415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4082140415},{"type":"v1","addr":"172.21.15.153:6805","nonce":4082140415}]},"public_addr":"172.21.15.153:6801/4082140415","cluster_addr":"172.21.15.153:6803/4082140415","heartbeat_back_addr":"172.21.15.153:6807/4082140415","heartbeat_front_addr":"172.21.15.153:6805/4082140415","state":["exists","up"]},{"osd":2,"uuid":"1d3e8e12-5a26-4f1b-acd8-82dbe93f92fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6801","nonce":2055802330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6803","nonce":2055802330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6807","nonce":2055802330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2055802330},{"type":"v1","addr":"172.21.15.173:6805","nonce":2055802330}]},"public_addr":"172.21.15.173:6801/2055802330","cluster_addr":"172.21.15.173:6803/2055802330","heartbeat_back_addr":"172.21.15.173:6807/2055802330","heartbeat_front_addr":"172.21.15.173:6805/2055802330","state":["exists","up"]},{"osd":3,"uuid":"5e1de00e-5fc6-4a53-9cfa-99cede3449aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6811","nonce":2502573186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6813","nonce":2502573186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6817","nonce":2502573186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2502573186},{"type":"v1","addr":"172.21.15.98:6815","nonce":2502573186}]},"public_addr":"172.21.15.98:6811/2502573186","cluster_addr":"172.21.15.98:6813/2502573186","heartbeat_back_addr":"172.21.15.98:6817/2502573186","heartbeat_front_addr":"172.21.15.98:6815/2502573186","state":["exists","up"]},{"osd":4,"uuid":"b864c756-758e-4799-a35d-b83d204b3727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6809","nonce":3184279467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6811","nonce":3184279467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6815","nonce":3184279467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3184279467},{"type":"v1","addr":"172.21.15.153:6813","nonce":3184279467}]},"public_addr":"172.21.15.153:6809/3184279467","cluster_addr":"172.21.15.153:6811/3184279467","heartbeat_back_addr":"172.21.15.153:6815/3184279467","heartbeat_front_addr":"172.21.15.153:6813/3184279467","state":["exists","up"]},{"osd":5,"uuid":"c31daad0-fd36-4b7c-98d3-64e2057c4779","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6809","nonce":3808045803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6811","nonce":3808045803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6815","nonce":3808045803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3808045803},{"type":"v1","addr":"172.21.15.173:6813","nonce":3808045803}]},"public_addr":"172.21.15.173:6809/3808045803","cluster_addr":"172.21.15.173:6811/3808045803","heartbeat_back_addr":"172.21.15.173:6815/3808045803","heartbeat_front_addr":"172.21.15.173:6813/3808045803","state":["exists","up"]},{"osd":6,"uuid":"229b1cd5-870e-4b6c-b3af-243b19a68157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3005930753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3005930753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6825","nonce":3005930753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3005930753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3005930753}]},"public_addr":"172.21.15.98:6819/3005930753","cluster_addr":"172.21.15.98:6821/3005930753","heartbeat_back_addr":"172.21.15.98:6825/3005930753","heartbeat_front_addr":"172.21.15.98:6823/3005930753","state":["exists","up"]},{"osd":7,"uuid":"615a84ba-ee1b-4ea0-92c9-0211098d2215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6817","nonce":838772754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6819","nonce":838772754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6823","nonce":838772754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":838772754},{"type":"v1","addr":"172.21.15.153:6821","nonce":838772754}]},"public_addr":"172.21.15.153:6817/838772754","cluster_addr":"172.21.15.153:6819/838772754","heartbeat_back_addr":"172.21.15.153:6823/838772754","heartbeat_front_addr":"172.21.15.153:6821/838772754","state":["exists","up"]},{"osd":8,"uuid":"fa255236-2797-4a72-b784-65cc6c731190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6817","nonce":785354592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6819","nonce":785354592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6823","nonce":785354592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":785354592},{"type":"v1","addr":"172.21.15.173:6821","nonce":785354592}]},"public_addr":"172.21.15.173:6817/785354592","cluster_addr":"172.21.15.173:6819/785354592","heartbeat_back_addr":"172.21.15.173:6823/785354592","heartbeat_front_addr":"172.21.15.173:6821/785354592","state":["exists","up"]},{"osd":9,"uuid":"cc2c29f0-167c-40d8-8ca6-4758e66f435c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6827","nonce":3340262807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6829","nonce":3340262807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6833","nonce":3340262807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3340262807},{"type":"v1","addr":"172.21.15.98:6831","nonce":3340262807}]},"public_addr":"172.21.15.98:6827/3340262807","cluster_addr":"172.21.15.98:6829/3340262807","heartbeat_back_addr":"172.21.15.98:6833/3340262807","heartbeat_front_addr":"172.21.15.98:6831/3340262807","state":["exists","up"]},{"osd":10,"uuid":"deab615a-9d50-4480-ac75-93033b168f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6825","nonce":2757700029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6827","nonce":2757700029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6831","nonce":2757700029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2757700029},{"type":"v1","addr":"172.21.15.153:6829","nonce":2757700029}]},"public_addr":"172.21.15.153:6825/2757700029","cluster_addr":"172.21.15.153:6827/2757700029","heartbeat_back_addr":"172.21.15.153:6831/2757700029","heartbeat_front_addr":"172.21.15.153:6829/2757700029","state":["exists","up"]},{"osd":11,"uuid":"95cf6dc5-aeb4-45b1-8812-3f2f2b7245ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6825","nonce":567612982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6827","nonce":567612982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6831","nonce":567612982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":567612982},{"type":"v1","addr":"172.21.15.173:6829","nonce":567612982}]},"public_addr":"172.21.15.173:6825/567612982","cluster_addr":"172.21.15.173:6827/567612982","heartbeat_back_addr":"172.21.15.173:6831/567612982","heartbeat_front_addr":"172.21.15.173:6829/567612982","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:33.213081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:12:58.600066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:22.856993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:13:55.263605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:21.655995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:14:46.841285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:21.062937+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:15:47.898989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:13.323674+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:16:47.267630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:14.895952+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T12:17:40.658063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.3","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6800/3396936170":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/352508062":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/653650228":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/719012091":"2024-03-17T12:10:42.632912+0000","172.21.15.98:6801/3029136477":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/196846375":"2024-03-17T12:10:42.632912+0000","172.21.15.98:0/3496387635":"2024-03-17T12:10:08.286891+0000","172.21.15.98:0/3016831032":"2024-03-17T12:10:08.286891+0000","172.21.15.98:6800/3029136477":"2024-03-17T12:10:08.286891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:25:16.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.183+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 msgr2=0x7f70180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.183+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 0x7f70180a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f703419af90 tx=0x7f7020005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:16.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.183+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 msgr2=0x7f7034199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.183+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f702400d950 tx=0x7f702400de10 comp rx=0 tx=0).stop 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 shutdown_connections 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f70180a2880 0x7f70180a4d40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034105390 0x7f70341a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70341049d0 0x7f7034199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 --2- 172.21.15.98:0/66366954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70341037e0 0x7f7034199a80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 >> 172.21.15.98:0/66366954 conn(0x7f70340fed50 msgr2=0x7f7034107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 shutdown_connections 2024-03-16T12:25:16.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.184+0000 7f7038e00640 1 -- 172.21.15.98:0/66366954 wait complete. 2024-03-16T12:25:16.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T12:25:16.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:16 smithi098 ceph-mon[31633]: pgmap v517: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:25:16.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:16 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4168553726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:16 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/165176047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mon[36632]: pgmap v517: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:25:16.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4168553726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/165176047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 -- 172.21.15.98:0/2127977000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 msgr2=0x7f69dc1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 --2- 172.21.15.98:0/2127977000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc1097f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009be0 tx=0x7f69d402bef0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 -- 172.21.15.98:0/2127977000 shutdown_connections 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 --2- 172.21.15.98:0/2127977000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc1097f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 --2- 172.21.15.98:0/2127977000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc104a70 0x7f69dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 --2- 172.21.15.98:0/2127977000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc102140 0x7f69dc104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 -- 172.21.15.98:0/2127977000 >> 172.21.15.98:0/2127977000 conn(0x7f69dc0fbb50 msgr2=0x7f69dc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 -- 172.21.15.98:0/2127977000 shutdown_connections 2024-03-16T12:25:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.398+0000 7f69e0c00640 1 -- 172.21.15.98:0/2127977000 wait complete. 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 Processor -- start 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 -- start start 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 0x7f69dc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d390 con 0x7f69dc102140 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d210 con 0x7f69dc1073a0 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69e0c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10d510 con 0x7f69dc104a70 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69db400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 0x7f69dc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69dbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69d3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39114/0 (socket says 172.21.15.98:39114) 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.399+0000 7f69d3e00640 1 -- 172.21.15.98:0/948767428 learned_addr learned my addr 172.21.15.98:0/948767428 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 -- 172.21.15.98:0/948767428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 msgr2=0x7f69dc197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 -- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 msgr2=0x7f69dc1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 0x7f69dc1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 -- 172.21.15.98:0/948767428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69dc19f580 con 0x7f69dc1073a0 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69d3e00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69db400640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 0x7f69dc1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:25:16.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69dbe00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc19ef10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f69d402c430 tx=0x7f69d40337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4043020 con 0x7f69dc1073a0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69d402ca60 con 0x7f69dc1073a0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.400+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4034690 con 0x7f69dc1073a0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.401+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69dc19f810 con 0x7f69dc1073a0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.401+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69dc19fdb0 con 0x7f69dc1073a0 2024-03-16T12:25:16.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.402+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69dc0fd640 con 0x7f69dc1073a0 2024-03-16T12:25:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.403+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f69d4046020 con 0x7f69dc1073a0 2024-03-16T12:25:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.405+0000 7f69d8c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 0x7f69c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.405+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69d40f13b0 con 0x7f69dc1073a0 2024-03-16T12:25:16.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.405+0000 7f69db400640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 0x7f69c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.405+0000 7f69db400640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 0x7f69c00a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0095f0 tx=0x7f69cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:16.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.409+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69d404d020 con 0x7f69dc1073a0 2024-03-16T12:25:16.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-mon[36901]: pgmap v517: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T12:25:16.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4168553726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/165176047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:25:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.654+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f69dc0623e0 con 0x7f69dc1073a0 2024-03-16T12:25:16.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.663+0000 7f69d8c00640 1 -- 172.21.15.98:0/948767428 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) ==== 161+0+0 (secure 0 0 0) 0x7f69d40be090 con 0x7f69dc1073a0 2024-03-16T12:25:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 msgr2=0x7f69c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 0x7f69c00a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0095f0 tx=0x7f69cc005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 msgr2=0x7f69dc19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc19ef10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f69d402c430 tx=0x7f69d40337b0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 shutdown_connections 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f69c00a2880 0x7f69c00a4d40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc1073a0 0x7f69dc19ef10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69dc104a70 0x7f69dc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 --2- 172.21.15.98:0/948767428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69dc102140 0x7f69dc1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 >> 172.21.15.98:0/948767428 conn(0x7f69dc0fbb50 msgr2=0x7f69dc102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 shutdown_connections 2024-03-16T12:25:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.667+0000 7f69e0c00640 1 -- 172.21.15.98:0/948767428 wait complete. 2024-03-16T12:25:16.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T12:25:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 -- 172.21.15.98:0/1674546419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41017a0 msgr2=0x7fabe4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 --2- 172.21.15.98:0/1674546419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41017a0 0x7fabe4101c20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009900 tx=0x7fabd002bc30 comp rx=0 tx=0).stop 2024-03-16T12:25:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 -- 172.21.15.98:0/1674546419 shutdown_connections 2024-03-16T12:25:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 --2- 172.21.15.98:0/1674546419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe4102160 0x7fabe41089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 --2- 172.21.15.98:0/1674546419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41017a0 0x7fabe4101c20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 --2- 172.21.15.98:0/1674546419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe41005b0 0x7fabe41009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 -- 172.21.15.98:0/1674546419 >> 172.21.15.98:0/1674546419 conn(0x7fabe40fbb20 msgr2=0x7fabe40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 -- 172.21.15.98:0/1674546419 shutdown_connections 2024-03-16T12:25:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.867+0000 7fabe9200640 1 -- 172.21.15.98:0/1674546419 wait complete. 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.868+0000 7fabe9200640 1 Processor -- start 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.868+0000 7fabe9200640 1 -- start start 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.868+0000 7fabe9200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.868+0000 7fabe9200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe41017a0 0x7fabe4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe9200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe4102160 0x7fabe41030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe9200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410c280 con 0x7fabe4102160 2024-03-16T12:25:16.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe9200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410c100 con 0x7fabe41005b0 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe9200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410c400 con 0x7fabe41017a0 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe4102160 0x7fabe41030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:49134/0 (socket says 172.21.15.98:49134) 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 -- 172.21.15.98:0/4182919047 learned_addr learned my addr 172.21.15.98:0/4182919047 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe2a00640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe41017a0 0x7fabe4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 -- 172.21.15.98:0/4182919047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe41017a0 msgr2=0x7fabe4102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a[31628]: 2024-03-16T12:25:16.655+0000 7fa443200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:25:16.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe41017a0 0x7fabe4102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 -- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe4102160 msgr2=0x7fabe41030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:16.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe4102160 0x7fabe41030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.869+0000 7fabe3400640 1 -- 172.21.15.98:0/4182919047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe4103950 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.870+0000 7fabe3400640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fabd400e9c0 tx=0x7fabd400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.870+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd400cde0 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.870+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd4010910 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.870+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe4197620 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.871+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd40189c0 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.871+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe4197b40 con 0x7fabe41005b0 2024-03-16T12:25:16.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.872+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe4197e00 con 0x7fabe41005b0 2024-03-16T12:25:16.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.874+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fabd4010ac0 con 0x7fabe41005b0 2024-03-16T12:25:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.876+0000 7fabdbe00640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 0x7fabc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:16.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.876+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fabd40cfb90 con 0x7fabe41005b0 2024-03-16T12:25:16.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.876+0000 7fabe2a00640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 0x7fabc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:16.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.877+0000 7fabe2a00640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 0x7fabc80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c240 tx=0x7fabd0043040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:16.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:16.881+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fabd4026070 con 0x7fabe41005b0 2024-03-16T12:25:17.002 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.002 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.002 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: starting mds.b at 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-mds[67036]: -- 172.21.15.153:0/4153383526 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597a8e83a40 con 0x5597a8e56800 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: starting mds.e at 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-mds[67351]: -- 172.21.15.153:0/828270650 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c2a167da40 con 0x55c2a1651000 2024-03-16T12:25:17.003 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:25:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: starting mds.h at 2024-03-16T12:25:17.029 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.029 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.029 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mds[67516]: -- 172.21.15.173:0/1846461880 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f99ed7ba40 con 0x55f99ed4e800 2024-03-16T12:25:17.029 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: starting mds.i at 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mds[67208]: -- 172.21.15.173:0/4203905036 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563644befa40 con 0x563644bc3000 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: starting mds.c at 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: ignoring --setuser ceph since I am not root 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: ignoring --setgroup ceph since I am not root 2024-03-16T12:25:17.030 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-mds[66902]: -- 172.21.15.173:0/1532934723 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558eee841a40 con 0x558eee814800 2024-03-16T12:25:17.031 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:25:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: starting mds.f at 2024-03-16T12:25:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.121+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fabe4198730 con 0x7fabe41005b0 2024-03-16T12:25:17.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/66366954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/948767428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T12:25:17.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:17 smithi098 ceph-mon[31633]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/66366954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/948767428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:25:17.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:25:17.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T12:25:17.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:17 smithi173 ceph-mon[36632]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/66366954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/948767428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:25:17.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:25:17.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T12:25:17.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:17 smithi153 ceph-mon[36901]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:25:17.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.670+0000 7fabdbe00640 1 -- 172.21.15.98:0/4182919047 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7fabd409c770 con 0x7fabe41005b0 2024-03-16T12:25:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.673+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 msgr2=0x7fabc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.673+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 0x7fabc80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c240 tx=0x7fabd0043040 comp rx=0 tx=0).stop 2024-03-16T12:25:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.673+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 msgr2=0x7fabe4104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:17.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.673+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fabd400e9c0 tx=0x7fabd400ee80 comp rx=0 tx=0).stop 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 shutdown_connections 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7fabc80a2800 0x7fabc80a4cc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabe4102160 0x7fabe41030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabe41017a0 0x7fabe4102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 --2- 172.21.15.98:0/4182919047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe41005b0 0x7fabe4104540 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 >> 172.21.15.98:0/4182919047 conn(0x7fabe40fbb20 msgr2=0x7fabe40fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 shutdown_connections 2024-03-16T12:25:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.675+0000 7fabe9200640 1 -- 172.21.15.98:0/4182919047 wait complete. 2024-03-16T12:25:17.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.895+0000 7efc37600640 1 -- 172.21.15.98:0/3955502695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30105390 msgr2=0x7efc30105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.895+0000 7efc37600640 1 --2- 172.21.15.98:0/3955502695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30105390 0x7efc30105810 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efc20009860 tx=0x7efc2002bb70 comp rx=0 tx=0).stop 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 -- 172.21.15.98:0/3955502695 shutdown_connections 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 --2- 172.21.15.98:0/3955502695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30105390 0x7efc30105810 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 --2- 172.21.15.98:0/3955502695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc301029e0 0x7efc30104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 --2- 172.21.15.98:0/3955502695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc30100070 0x7efc301024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 -- 172.21.15.98:0/3955502695 >> 172.21.15.98:0/3955502695 conn(0x7efc300fbc70 msgr2=0x7efc300fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 -- 172.21.15.98:0/3955502695 shutdown_connections 2024-03-16T12:25:17.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.896+0000 7efc37600640 1 -- 172.21.15.98:0/3955502695 wait complete. 2024-03-16T12:25:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 Processor -- start 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 -- start start 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 0x7efc3019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc30105390 0x7efc301a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc30113a50 con 0x7efc30105390 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc301138d0 con 0x7efc30100070 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc37600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc30113bd0 con 0x7efc301029e0 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc34c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc35600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc30105390 0x7efc301a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc34c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.98:49140/0 (socket says 172.21.15.98:49140) 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.897+0000 7efc34c00640 1 -- 172.21.15.98:0/593512854 learned_addr learned my addr 172.21.15.98:0/593512854 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc2fe00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 0x7efc3019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 -- 172.21.15.98:0/593512854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 msgr2=0x7efc3019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 0x7efc3019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 -- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc30105390 msgr2=0x7efc301a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc30105390 0x7efc301a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 -- 172.21.15.98:0/593512854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc301a5ce0 con 0x7efc30100070 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc2fe00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 0x7efc3019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc34c00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efc28009d50 tx=0x7efc2800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28003b70 con 0x7efc30100070 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc28003d10 con 0x7efc30100070 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28010a20 con 0x7efc30100070 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc301a5fd0 con 0x7efc30100070 2024-03-16T12:25:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.898+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc301a6490 con 0x7efc30100070 2024-03-16T12:25:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.899+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc300fd760 con 0x7efc30100070 2024-03-16T12:25:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.901+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efc28003ec0 con 0x7efc30100070 2024-03-16T12:25:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.902+0000 7efc2d600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 0x7efc140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.903+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efc280cf570 con 0x7efc30100070 2024-03-16T12:25:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.903+0000 7efc2fe00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 0x7efc140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.903+0000 7efc2fe00640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 0x7efc140a4d40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc3019f5d0 tx=0x7efc18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:17.907+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc2809c1b0 con 0x7efc30100070 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: pgmap v519: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/4182919047' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: Cluster is now healthy 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6837/164773559,v1:172.21.15.153:6839/164773559] up:boot 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6834/1868360005,v1:172.21.15.173:6835/1868360005] up:boot 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6834/3917670884,v1:172.21.15.153:6835/3917670884] up:boot 2024-03-16T12:25:18.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.153:6836/109767822,v1:172.21.15.153:6838/109767822] up:boot 2024-03-16T12:25:18.035 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6836/1007190293,v1:172.21.15.173:6838/1007190293] up:boot 2024-03-16T12:25:18.035 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: mds.? [v2:172.21.15.173:6837/1022032809,v1:172.21.15.173:6839/1022032809] up:boot 2024-03-16T12:25:18.035 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: fsmap 12 up:standby 2024-03-16T12:25:18.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:18.095+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7efc30062480 con 0x7efc30100070 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:25:18.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:18 smithi173 ceph-mon[36632]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:25:18.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: pgmap v519: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/4182919047' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: Cluster is now healthy 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6837/164773559,v1:172.21.15.153:6839/164773559] up:boot 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6834/1868360005,v1:172.21.15.173:6835/1868360005] up:boot 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6834/3917670884,v1:172.21.15.153:6835/3917670884] up:boot 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.153:6836/109767822,v1:172.21.15.153:6838/109767822] up:boot 2024-03-16T12:25:18.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6836/1007190293,v1:172.21.15.173:6838/1007190293] up:boot 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: mds.? [v2:172.21.15.173:6837/1022032809,v1:172.21.15.173:6839/1022032809] up:boot 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: fsmap 12 up:standby 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:25:18.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:18 smithi098 ceph-mon[31633]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:25:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: pgmap v519: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T12:25:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/4182919047' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:25:18.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: Cluster is now healthy 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6837/164773559,v1:172.21.15.153:6839/164773559] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6834/1868360005,v1:172.21.15.173:6835/1868360005] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6834/3917670884,v1:172.21.15.153:6835/3917670884] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.153:6836/109767822,v1:172.21.15.153:6838/109767822] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6836/1007190293,v1:172.21.15.173:6838/1007190293] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: mds.? [v2:172.21.15.173:6837/1022032809,v1:172.21.15.173:6839/1022032809] up:boot 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: fsmap 12 up:standby 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:25:18.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:25:18.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:18 smithi153 ceph-mon[36901]: from='mgr.14152 172.21.15.98:0/1394832741' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:25:19.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.069+0000 7efc2d600640 1 -- 172.21.15.98:0/593512854 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) ==== 181+0+0 (secure 0 0 0) 0x7efc2809bab0 con 0x7efc30100070 2024-03-16T12:25:19.051 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2024-03-16T12:25:19.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 msgr2=0x7efc140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:19.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 0x7efc140a4d40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc3019f5d0 tx=0x7efc18009260 comp rx=0 tx=0).stop 2024-03-16T12:25:19.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 msgr2=0x7efc3019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:19.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efc28009d50 tx=0x7efc2800c6a0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 shutdown_connections 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7efc140a2880 0x7efc140a4d40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc30105390 0x7efc301a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc301029e0 0x7efc3019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 --2- 172.21.15.98:0/593512854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc30100070 0x7efc3019e0c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.074+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 >> 172.21.15.98:0/593512854 conn(0x7efc300fbc70 msgr2=0x7efc301009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.075+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 shutdown_connections 2024-03-16T12:25:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.075+0000 7efc37600640 1 -- 172.21.15.98:0/593512854 wait complete. 2024-03-16T12:25:19.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T12:25:19.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 -- 172.21.15.98:0/4145298897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208104af0 msgr2=0x7f7208104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 --2- 172.21.15.98:0/4145298897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208104af0 0x7f7208104f70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009860 tx=0x7f71f002bb70 comp rx=0 tx=0).stop 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 -- 172.21.15.98:0/4145298897 shutdown_connections 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 --2- 172.21.15.98:0/4145298897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f720810bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 --2- 172.21.15.98:0/4145298897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208104af0 0x7f7208104f70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 --2- 172.21.15.98:0/4145298897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208103900 0x7f7208103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 -- 172.21.15.98:0/4145298897 >> 172.21.15.98:0/4145298897 conn(0x7f72080fee90 msgr2=0x7f72081012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 -- 172.21.15.98:0/4145298897 shutdown_connections 2024-03-16T12:25:19.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.319+0000 7f720cc00640 1 -- 172.21.15.98:0/4145298897 wait complete. 2024-03-16T12:25:19.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.320+0000 7f720cc00640 1 Processor -- start 2024-03-16T12:25:19.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.320+0000 7f720cc00640 1 -- start start 2024-03-16T12:25:19.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208103900 0x7f7208199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208104af0 0x7f720819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f720810f560 con 0x7f72081054b0 2024-03-16T12:25:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f720810f3e0 con 0x7f7208104af0 2024-03-16T12:25:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f720cc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f720810f6e0 con 0x7f7208103900 2024-03-16T12:25:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f7207e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f7207e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54608/0 (socket says 172.21.15.98:54608) 2024-03-16T12:25:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f7207e00640 1 -- 172.21.15.98:0/1311884116 learned_addr learned my addr 172.21.15.98:0/1311884116 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f7207400640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208103900 0x7f7208199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.321+0000 7f71ffe00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208104af0 0x7f720819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 -- 172.21.15.98:0/1311884116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208103900 msgr2=0x7f7208199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208103900 0x7f7208199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 -- 172.21.15.98:0/1311884116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208104af0 msgr2=0x7f720819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208104af0 0x7f720819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 -- 172.21.15.98:0/1311884116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72081a1860 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7207e00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f720000efc0 tx=0x7f720000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200003970 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7200004020 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72081a1b50 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720000b9f0 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.322+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72081a2050 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.324+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720810ac70 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.326+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f720001d020 con 0x7f72081054b0 2024-03-16T12:25:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.328+0000 7f7204c00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 0x7f71ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.328+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8854+0+0 (secure 0 0 0) 0x7f72000cec40 con 0x7f72081054b0 2024-03-16T12:25:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.328+0000 7f7207400640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 0x7f71ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.329+0000 7f7207400640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 0x7f71ec0a4cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f720819b120 tx=0x7f71f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.332+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f720009ba10 con 0x7f72081054b0 2024-03-16T12:25:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:19 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/593512854' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:19 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:19 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/593512854' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:19 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:19 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/593512854' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:19 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:19.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:19.530+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f72080623e0 con 0x7f72081054b0 2024-03-16T12:25:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.067+0000 7f7204c00640 1 -- 172.21.15.98:0/1311884116 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) ==== 169+0+0 (secure 0 0 0) 0x7f720009b310 con 0x7f72081054b0 2024-03-16T12:25:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.070+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 msgr2=0x7f71ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.070+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 0x7f71ec0a4cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f720819b120 tx=0x7f71f8005f90 comp rx=0 tx=0).stop 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.070+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 msgr2=0x7f72081a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.070+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f720000efc0 tx=0x7f720000c800 comp rx=0 tx=0).stop 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 shutdown_connections 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f71ec0a2800 0x7f71ec0a4cc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72081054b0 0x7f72081a11c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7208104af0 0x7f720819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 --2- 172.21.15.98:0/1311884116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7208103900 0x7f7208199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 >> 172.21.15.98:0/1311884116 conn(0x7f72080fee90 msgr2=0x7f7208107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 shutdown_connections 2024-03-16T12:25:20.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.071+0000 7f720cc00640 1 -- 172.21.15.98:0/1311884116 wait complete. 2024-03-16T12:25:20.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-16T12:25:20.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 -- 172.21.15.98:0/507458155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441029e0 msgr2=0x7f7f44102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:20.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/507458155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441029e0 0x7f7f44102e60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009860 tx=0x7f7f2c02bb70 comp rx=0 tx=0).stop 2024-03-16T12:25:20.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 -- 172.21.15.98:0/507458155 shutdown_connections 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/507458155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441033a0 0x7f7f44109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/507458155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441029e0 0x7f7f44102e60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/507458155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441017f0 0x7f7f44101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 -- 172.21.15.98:0/507458155 >> 172.21.15.98:0/507458155 conn(0x7f7f440fcd80 msgr2=0x7f7f440ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 -- 172.21.15.98:0/507458155 shutdown_connections 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.294+0000 7f7f4bc00640 1 -- 172.21.15.98:0/507458155 wait complete. 2024-03-16T12:25:20.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 Processor -- start 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 -- start start 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029e0 0x7f7f44198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441033a0 0x7f7f4419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4410d470 con 0x7f7f441017f0 2024-03-16T12:25:20.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4410d2f0 con 0x7f7f441029e0 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.295+0000 7f7f4bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4410d5f0 con 0x7f7f441033a0 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54618/0 (socket says 172.21.15.98:54618) 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 -- 172.21.15.98:0/2047983842 learned_addr learned my addr 172.21.15.98:0/2047983842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49c00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441033a0 0x7f7f4419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f3be00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029e0 0x7f7f44198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:20.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 -- 172.21.15.98:0/2047983842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441033a0 msgr2=0x7f7f4419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441033a0 0x7f7f4419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 -- 172.21.15.98:0/2047983842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029e0 msgr2=0x7f7f44198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029e0 0x7f7f44198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 -- 172.21.15.98:0/2047983842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4419f760 con 0x7f7f441017f0 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.296+0000 7f7f49200640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400eb70 tx=0x7f7f3400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.297+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34003b70 con 0x7f7f441017f0 2024-03-16T12:25:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.297+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3400ee70 con 0x7f7f441017f0 2024-03-16T12:25:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.297+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4419fa50 con 0x7f7f441017f0 2024-03-16T12:25:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.297+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34010830 con 0x7f7f441017f0 2024-03-16T12:25:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.297+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4419ff20 con 0x7f7f441017f0 2024-03-16T12:25:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.298+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f44108be0 con 0x7f7f441017f0 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.300+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7f34004020 con 0x7f7f441017f0 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.301+0000 7f7f42000640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 0x7f7f280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.301+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8324+0+0 (secure 0 0 0) 0x7f7f340cf130 con 0x7f7f441017f0 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.301+0000 7f7f3be00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 0x7f7f280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.302+0000 7f7f3be00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 0x7f7f280a4cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009830 tx=0x7f7f2c034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:25:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.305+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f3409c170 con 0x7f7f441017f0 2024-03-16T12:25:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:20 smithi173 ceph-mon[36632]: pgmap v520: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 716 B/s wr, 6 op/s 2024-03-16T12:25:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:20 smithi173 ceph-mon[36632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:20 smithi173 ceph-mon[36632]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T12:25:20.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:20 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:20 smithi098 ceph-mon[31633]: pgmap v520: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 716 B/s wr, 6 op/s 2024-03-16T12:25:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:20 smithi098 ceph-mon[31633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:20 smithi098 ceph-mon[31633]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T12:25:20.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:20 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:20.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:20.516+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7f440623e0 con 0x7f7f441017f0 2024-03-16T12:25:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:20 smithi153 ceph-mon[36901]: pgmap v520: 129 pgs: 129 active+clean; 1.2 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 716 B/s wr, 6 op/s 2024-03-16T12:25:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:20 smithi153 ceph-mon[36901]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:20 smithi153 ceph-mon[36901]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T12:25:20.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:20 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.092+0000 7f7f42000640 1 -- 172.21.15.98:0/2047983842 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v95) ==== 178+0+0 (secure 0 0 0) 0x7f7f3409ba70 con 0x7f7f441017f0 2024-03-16T12:25:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' removed 2024-03-16T12:25:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.097+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 msgr2=0x7f7f280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.097+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 0x7f7f280a4cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009830 tx=0x7f7f2c034ab0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.097+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 msgr2=0x7f7f44197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:25:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.097+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400eb70 tx=0x7f7f3400c6a0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 shutdown_connections 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:6800/3089653509,v1:172.21.15.98:6801/3089653509] conn(0x7f7f280a2800 0x7f7f280a4cc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f441033a0 0x7f7f4419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f441029e0 0x7f7f44198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 --2- 172.21.15.98:0/2047983842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f441017f0 0x7f7f44197b40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 >> 172.21.15.98:0/2047983842 conn(0x7f7f440fcd80 msgr2=0x7f7f44105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:25:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 shutdown_connections 2024-03-16T12:25:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-16T12:25:21.098+0000 7f7f4bc00640 1 -- 172.21.15.98:0/2047983842 wait complete. 2024-03-16T12:25:21.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:25:21.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:25:21.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:25:21.263 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T12:25:21.263 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:25:21.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:21 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:21.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:21 smithi173 ceph-mon[36632]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T12:25:21.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:21 smithi173 ceph-mon[36632]: from='client.? 172.21.15.98:0/2047983842' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:21.320 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:25:21.339 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:25:21.356 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T12:25:21.357 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T12:25:21.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a 2024-03-16T12:25:21.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:21.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 ceph-mon[31633]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T12:25:21.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 ceph-mon[31633]: from='client.? 172.21.15.98:0/2047983842' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:21.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:21 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/1311884116' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:25:21.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:21 smithi153 ceph-mon[36901]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T12:25:21.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:21 smithi153 ceph-mon[36901]: from='client.? 172.21.15.98:0/2047983842' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:25:21.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 systemd[1]: Stopping Ceph mon.a for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:22.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a[31628]: 2024-03-16T12:25:21.953+0000 7fa44a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:22.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a[31628]: 2024-03-16T12:25:21.953+0000 7fa44a000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T12:25:22.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:22 smithi098 podman[91540]: 2024-03-16 12:25:22.242820007 +0000 UTC m=+0.430162593 container died 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T12:25:22.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:22 smithi098 podman[91540]: 2024-03-16 12:25:22.840104707 +0000 UTC m=+1.027447299 container cleanup 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T12:25:22.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:22 smithi098 bash[91540]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a 2024-03-16T12:25:23.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 16 12:25:23 smithi098 podman[91554]: 2024-03-16 12:25:23.125597398 +0000 UTC m=+0.863604163 container remove 10511c15947855f51145093479609dbf817bb9a7c06891267b3a3e9a6c9f9824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:25:23.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T12:25:23.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:23.681 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T12:25:23.682 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T12:25:23.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.b 2024-03-16T12:25:24.002 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:23 smithi153 systemd[1]: Stopping Ceph mon.b for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:24.478 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b[36897]: 2024-03-16T12:25:24.022+0000 7f41377f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:24.478 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b[36897]: 2024-03-16T12:25:24.022+0000 7f41377f3700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T12:25:24.478 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:24 smithi153 podman[70666]: 2024-03-16 12:25:24.186075891 +0000 UTC m=+0.287450129 container died 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T12:25:24.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:24 smithi153 podman[70666]: 2024-03-16 12:25:24.589550363 +0000 UTC m=+0.690924611 container cleanup 254ec67692aa6be259e03798d59b52acd3719115cdcaeca76689f3c022e4cf6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True) 2024-03-16T12:25:24.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 16 12:25:24 smithi153 bash[70666]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-b 2024-03-16T12:25:24.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T12:25:25.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:25.029 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T12:25:25.029 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T12:25:25.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.c 2024-03-16T12:25:25.318 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 systemd[1]: Stopping Ceph mon.c for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:25.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c[36613]: 2024-03-16T12:25:25.373+0000 7faa9bc44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:25.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c[36613]: 2024-03-16T12:25:25.374+0000 7faa9bc44700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T12:25:25.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 podman[70599]: 2024-03-16 12:25:25.568854522 +0000 UTC m=+0.324881595 container died 7eb0a02b24449406021362f4813dd597a6bb7f079d4d2d28207a63e5b2fbe6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:25:26.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 podman[70599]: 2024-03-16 12:25:25.9592423 +0000 UTC m=+0.715269372 container cleanup 7eb0a02b24449406021362f4813dd597a6bb7f079d4d2d28207a63e5b2fbe6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:25:26.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Mar 16 12:25:25 smithi173 bash[70599]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mon-c 2024-03-16T12:25:26.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T12:25:26.400 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:26.400 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T12:25:26.400 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T12:25:26.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x 2024-03-16T12:25:26.741 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:26 smithi098 systemd[1]: Stopping Ceph mgr.x for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:27.431 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:25:27.010+0000 7fcf47200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:27.432 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x[31872]: 2024-03-16T12:25:27.010+0000 7fcf47200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T12:25:27.432 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 podman[91660]: 2024-03-16 12:25:27.104735869 +0000 UTC m=+0.266843878 container died 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T12:25:27.991 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 podman[91660]: 2024-03-16 12:25:27.747446255 +0000 UTC m=+0.909554286 container cleanup 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T12:25:27.991 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 bash[91660]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x 2024-03-16T12:25:27.991 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Mar 16 12:25:27 smithi098 podman[91675]: 2024-03-16 12:25:27.9746384 +0000 UTC m=+0.860065306 container remove 1661ffdd4ffff5a65e57d80380bfabeb01cdc457fff9e80488aced0ac58a042d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T12:25:28.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T12:25:28.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:28.589 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T12:25:28.590 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T12:25:28.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y 2024-03-16T12:25:28.911 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:28 smithi153 systemd[1]: Stopping Ceph mgr.y for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:29.243 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:28 smithi153 podman[70787]: 2024-03-16 12:25:28.93227769 +0000 UTC m=+0.124888690 container died 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T12:25:29.502 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 podman[70787]: 2024-03-16 12:25:29.370441216 +0000 UTC m=+0.563052205 container cleanup 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:25:29.502 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 bash[70787]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y 2024-03-16T12:25:29.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T12:25:29.833 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 podman[70801]: 2024-03-16 12:25:29.588697236 +0000 UTC m=+0.654568621 container remove 87b50ce54ad14c3f5b887ada36d860e3266b613ada80c18a382926294870e99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T12:25:29.833 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-16T12:25:29.833 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-16T12:25:29.833 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 systemd[1]: Stopped Ceph mgr.y for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:25:29.833 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Mar 16 12:25:29 smithi153 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.y.service: Consumed 10.777s CPU time. 2024-03-16T12:25:29.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:29.846 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T12:25:29.846 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T12:25:29.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.z 2024-03-16T12:25:30.206 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:25:29 smithi173 systemd[1]: Stopping Ceph mgr.z for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:30.467 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:25:30 smithi173 podman[70719]: 2024-03-16 12:25:30.227268222 +0000 UTC m=+0.167475426 container died 94b176253d9e0449e947df08fe3787198f7e2b1b548e71ab3f12ea4975142506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:25:30.818 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:25:30 smithi173 podman[70719]: 2024-03-16 12:25:30.598949401 +0000 UTC m=+0.539156598 container cleanup 94b176253d9e0449e947df08fe3787198f7e2b1b548e71ab3f12ea4975142506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-16T12:25:30.818 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Mar 16 12:25:30 smithi173 bash[70719]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mgr-z 2024-03-16T12:25:31.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T12:25:31.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:31.191 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T12:25:31.191 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T12:25:31.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.0 2024-03-16T12:25:31.706 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:31 smithi098 systemd[1]: Stopping Ceph osd.0 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:31.991 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0[46633]: 2024-03-16T12:25:31.725+0000 7fa040600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:31.991 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0[46633]: 2024-03-16T12:25:31.725+0000 7fa040600700 -1 osd.0 95 *** Got signal Terminated *** 2024-03-16T12:25:31.991 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0[46633]: 2024-03-16T12:25:31.725+0000 7fa040600700 -1 osd.0 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:25:37.491 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:37 smithi098 podman[91781]: 2024-03-16 12:25:37.202592275 +0000 UTC m=+5.626516249 container stop c66d1a4f17b5691813b40ac785f7439025ac332a3eb0d9f809e728f929d873f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T12:25:37.492 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:37 smithi098 podman[91781]: 2024-03-16 12:25:37.228671614 +0000 UTC m=+5.652595589 container died c66d1a4f17b5691813b40ac785f7439025ac332a3eb0d9f809e728f929d873f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T12:25:37.819 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:37 smithi098 podman[91781]: 2024-03-16 12:25:37.82030585 +0000 UTC m=+6.244229854 container cleanup c66d1a4f17b5691813b40ac785f7439025ac332a3eb0d9f809e728f929d873f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:25:37.819 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:37 smithi098 bash[91781]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0 2024-03-16T12:25:38.491 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:38 smithi098 podman[91800]: 2024-03-16 12:25:38.120825377 +0000 UTC m=+0.901004750 container remove c66d1a4f17b5691813b40ac785f7439025ac332a3eb0d9f809e728f929d873f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:25:39.161 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:38 smithi098 podman[91890]: 2024-03-16 12:25:38.835447597 +0000 UTC m=+0.034794301 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:25:39.161 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:38 smithi098 podman[91890]: 2024-03-16 12:25:38.956778569 +0000 UTC m=+0.156125310 container create 18ffcf1461e37de945097977cffa75ae185e8b8b53584f5835a18fbe0ecb2dfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T12:25:39.491 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:39 smithi098 podman[91890]: 2024-03-16 12:25:39.302617039 +0000 UTC m=+0.501963787 container init 18ffcf1461e37de945097977cffa75ae185e8b8b53584f5835a18fbe0ecb2dfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-16T12:25:39.491 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:39 smithi098 podman[91890]: 2024-03-16 12:25:39.318919886 +0000 UTC m=+0.518266647 container start 18ffcf1461e37de945097977cffa75ae185e8b8b53584f5835a18fbe0ecb2dfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:25:39.491 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:39 smithi098 podman[91890]: 2024-03-16 12:25:39.418318564 +0000 UTC m=+0.617665342 container attach 18ffcf1461e37de945097977cffa75ae185e8b8b53584f5835a18fbe0ecb2dfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:25:39.991 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 16 12:25:39 smithi098 podman[91890]: 2024-03-16 12:25:39.750824621 +0000 UTC m=+0.950171369 container died 18ffcf1461e37de945097977cffa75ae185e8b8b53584f5835a18fbe0ecb2dfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T12:25:40.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T12:25:40.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:40.739 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T12:25:40.739 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T12:25:40.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.1 2024-03-16T12:25:41.053 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:40 smithi153 systemd[1]: Stopping Ceph osd.1 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:41.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1[41485]: 2024-03-16T12:25:41.073+0000 7f4560660700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:41.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1[41485]: 2024-03-16T12:25:41.073+0000 7f4560660700 -1 osd.1 95 *** Got signal Terminated *** 2024-03-16T12:25:41.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1[41485]: 2024-03-16T12:25:41.073+0000 7f4560660700 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:25:46.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:46 smithi153 podman[70924]: 2024-03-16 12:25:46.547254974 +0000 UTC m=+5.590716700 container died e665ef106a918781b222545edd0bcbbd71d3ea1962b3f24ae45f302c3cea5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:25:47.196 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:46 smithi153 podman[70924]: 2024-03-16 12:25:46.949698708 +0000 UTC m=+5.993160431 container cleanup e665ef106a918781b222545edd0bcbbd71d3ea1962b3f24ae45f302c3cea5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-16T12:25:47.196 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:46 smithi153 bash[70924]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1 2024-03-16T12:25:47.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[70940]: 2024-03-16 12:25:47.217777572 +0000 UTC m=+0.665840053 container remove e665ef106a918781b222545edd0bcbbd71d3ea1962b3f24ae45f302c3cea5025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T12:25:47.816 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[71031]: 2024-03-16 12:25:47.46800862 +0000 UTC m=+0.018326489 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:25:47.817 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[71031]: 2024-03-16 12:25:47.585381312 +0000 UTC m=+0.135699167 container create 18bbddfd4b8d2542a0106b256bd7fdac6ecd3b339df6316d2108cf1f77c4bb60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-16T12:25:48.108 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[71031]: 2024-03-16 12:25:47.838035562 +0000 UTC m=+0.388353416 container init 18bbddfd4b8d2542a0106b256bd7fdac6ecd3b339df6316d2108cf1f77c4bb60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-16T12:25:48.108 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[71031]: 2024-03-16 12:25:47.841492846 +0000 UTC m=+0.391810705 container start 18bbddfd4b8d2542a0106b256bd7fdac6ecd3b339df6316d2108cf1f77c4bb60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:25:48.108 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:47 smithi153 podman[71031]: 2024-03-16 12:25:47.963670183 +0000 UTC m=+0.513988040 container attach 18bbddfd4b8d2542a0106b256bd7fdac6ecd3b339df6316d2108cf1f77c4bb60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T12:25:48.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 16 12:25:48 smithi153 podman[71031]: 2024-03-16 12:25:48.129374702 +0000 UTC m=+0.679692586 container died 18bbddfd4b8d2542a0106b256bd7fdac6ecd3b339df6316d2108cf1f77c4bb60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T12:25:48.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T12:25:48.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:48.847 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T12:25:48.847 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T12:25:48.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.2 2024-03-16T12:25:49.174 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:48 smithi173 systemd[1]: Stopping Ceph osd.2 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:49.568 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2[41753]: 2024-03-16T12:25:49.194+0000 7f7a23c12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:49.568 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2[41753]: 2024-03-16T12:25:49.194+0000 7f7a23c12700 -1 osd.2 95 *** Got signal Terminated *** 2024-03-16T12:25:49.568 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2[41753]: 2024-03-16T12:25:49.194+0000 7f7a23c12700 -1 osd.2 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:25:54.767 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:54 smithi173 podman[70854]: 2024-03-16 12:25:54.522603112 +0000 UTC m=+5.458281043 container died cbeee5bfbc6007db3e868745c349613338d5ed457dbd1058291fb630ed2734bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T12:25:55.068 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:54 smithi173 podman[70854]: 2024-03-16 12:25:54.895734377 +0000 UTC m=+5.831412296 container cleanup cbeee5bfbc6007db3e868745c349613338d5ed457dbd1058291fb630ed2734bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T12:25:55.068 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:54 smithi173 bash[70854]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2 2024-03-16T12:25:55.481 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70872]: 2024-03-16 12:25:55.180614393 +0000 UTC m=+0.659201561 container remove cbeee5bfbc6007db3e868745c349613338d5ed457dbd1058291fb630ed2734bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:25:55.789 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70968]: 2024-03-16 12:25:55.405591454 +0000 UTC m=+0.017474742 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:25:55.789 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70968]: 2024-03-16 12:25:55.506596202 +0000 UTC m=+0.118479494 container create 3b2b7906e7d57da16b227bd1243966e3a743bf73eaeb058ba72125fa88ba709d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:25:55.789 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70968]: 2024-03-16 12:25:55.735606824 +0000 UTC m=+0.347490121 container init 3b2b7906e7d57da16b227bd1243966e3a743bf73eaeb058ba72125fa88ba709d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T12:25:55.789 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70968]: 2024-03-16 12:25:55.739045992 +0000 UTC m=+0.350929282 container start 3b2b7906e7d57da16b227bd1243966e3a743bf73eaeb058ba72125fa88ba709d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T12:25:55.789 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:55 smithi173 podman[70968]: 2024-03-16 12:25:55.810296603 +0000 UTC m=+0.422179903 container attach 3b2b7906e7d57da16b227bd1243966e3a743bf73eaeb058ba72125fa88ba709d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T12:25:56.068 INFO:journalctl@ceph.osd.2.smithi173.stdout:Mar 16 12:25:56 smithi173 podman[70968]: 2024-03-16 12:25:56.027993792 +0000 UTC m=+0.639877075 container died 3b2b7906e7d57da16b227bd1243966e3a743bf73eaeb058ba72125fa88ba709d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-16T12:25:56.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T12:25:56.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:25:56.675 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T12:25:56.675 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T12:25:56.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.3 2024-03-16T12:25:57.210 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:25:56 smithi098 systemd[1]: Stopping Ceph osd.3 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:25:57.492 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:25:57 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3[53338]: 2024-03-16T12:25:57.297+0000 7ff0e4e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:25:57.492 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:25:57 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3[53338]: 2024-03-16T12:25:57.297+0000 7ff0e4e00700 -1 osd.3 95 *** Got signal Terminated *** 2024-03-16T12:25:57.492 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:25:57 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3[53338]: 2024-03-16T12:25:57.297+0000 7ff0e4e00700 -1 osd.3 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:25:58.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:25:58 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:25:58.181+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:25:59.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:25:59 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:25:59.166+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:00.318 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:00.062+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:00.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:00 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:00.215+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:00.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:00.159+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:01.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:01.144+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:01.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:01.096+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:01.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:01 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:01.265+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:02.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:02 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:02.142+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:02.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:02.099+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:02.722 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:02 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:02.262+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:02.990 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:02 smithi098 podman[91993]: 2024-03-16 12:26:02.742707299 +0000 UTC m=+5.593325158 container died 8ffe887bc0a70b5d49056ee9d317a7e8422c586f134079616f4399604c3c0387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T12:26:03.295 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:03 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:03.271+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:03.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:03 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:03.127+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:03.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:03.125+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:03.614 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:03 smithi098 podman[91993]: 2024-03-16 12:26:03.316689941 +0000 UTC m=+6.167307797 container cleanup 8ffe887bc0a70b5d49056ee9d317a7e8422c586f134079616f4399604c3c0387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:26:03.615 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:03 smithi098 bash[91993]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3 2024-03-16T12:26:03.990 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:03 smithi098 podman[92024]: 2024-03-16 12:26:03.635164581 +0000 UTC m=+0.878533267 container remove 8ffe887bc0a70b5d49056ee9d317a7e8422c586f134079616f4399604c3c0387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T12:26:04.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:04 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:04.253+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:04.491 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:04 smithi098 podman[92112]: 2024-03-16 12:26:04.202175257 +0000 UTC m=+0.035135801 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:04.491 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:04 smithi098 podman[92112]: 2024-03-16 12:26:04.327991832 +0000 UTC m=+0.160952407 container create 78e703b1e5336525fb5c7fdc08bec69147849a210b86cc5f22fcbdfc3292f95f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T12:26:04.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:04 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:04.120+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:04.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:04.150+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:04.991 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:04 smithi098 podman[92112]: 2024-03-16 12:26:04.640420895 +0000 UTC m=+0.473381487 container init 78e703b1e5336525fb5c7fdc08bec69147849a210b86cc5f22fcbdfc3292f95f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:04.991 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:04 smithi098 podman[92112]: 2024-03-16 12:26:04.651414523 +0000 UTC m=+0.484375104 container start 78e703b1e5336525fb5c7fdc08bec69147849a210b86cc5f22fcbdfc3292f95f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T12:26:04.991 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:04 smithi098 podman[92112]: 2024-03-16 12:26:04.74771093 +0000 UTC m=+0.580671535 container attach 78e703b1e5336525fb5c7fdc08bec69147849a210b86cc5f22fcbdfc3292f95f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-16T12:26:05.442 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:05 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:05.218+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:05.442 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 16 12:26:05 smithi098 podman[92112]: 2024-03-16 12:26:05.082188105 +0000 UTC m=+0.915148678 container died 78e703b1e5336525fb5c7fdc08bec69147849a210b86cc5f22fcbdfc3292f95f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T12:26:05.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:05 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:05.152+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:05.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:05.188+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:05.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T12:26:06.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:06.050 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T12:26:06.050 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T12:26:06.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.4 2024-03-16T12:26:06.356 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:06 smithi153 systemd[1]: Stopping Ceph osd.4 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:06.357 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:06.169+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:06.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:06 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:06.208+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:06.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:06.238+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:06.751 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:06.377+0000 7f67f953c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:06.752 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:06.377+0000 7f67f953c700 -1 osd.4 95 *** Got signal Terminated *** 2024-03-16T12:26:06.752 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:06 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:06.377+0000 7f67f953c700 -1 osd.4 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:07.067 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:06.793+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:07.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:07.180+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:07.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:07 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:07.178+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:07.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:07.274+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:07.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:07 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:07.550+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:08.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:07 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:07.580+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:08.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:07.804+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:08.068 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:07.982+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:08.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:08 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:08.228+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:08.501 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:08.350+0000 7f67f1b28700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.883209+0000 front 2024-03-16T12:25:45.883187+0000 (oldest deadline 2024-03-16T12:26:07.583339+0000) 2024-03-16T12:26:08.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:08.227+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:08.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:08.283+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:08.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:08 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:08.534+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:09.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:08 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:08.550+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:09.067 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:08.998+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:09.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:08.815+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:09.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:09 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:09.204+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:09.495 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:09.373+0000 7f67f1b28700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.883209+0000 front 2024-03-16T12:25:45.883187+0000 (oldest deadline 2024-03-16T12:26:07.583339+0000) 2024-03-16T12:26:09.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:09.200+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:09.496 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:09.200+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:09.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:09.332+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:09.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:09 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:09.517+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:09.892 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:09 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:09.576+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:10.068 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:10 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:10.048+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:10.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:09.837+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:10.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:10 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:10.190+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:10.501 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:10.473+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:10.502 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4[48333]: 2024-03-16T12:26:10.388+0000 7f67f1b28700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.883209+0000 front 2024-03-16T12:25:45.883187+0000 (oldest deadline 2024-03-16T12:26:07.583339+0000) 2024-03-16T12:26:10.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:10.246+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:10.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:10 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:10.246+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:10.766 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:10 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:10.347+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:10.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:10 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:10.597+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:11.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:10 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:10.787+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:11.329 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:11 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:11.091+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:11.329 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:11 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:11.319+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:11.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:11 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:11.232+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:11.501 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:11 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:11.482+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:11.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:11 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:11.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:11.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:11 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:11.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:11.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:11 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:11.593+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:12.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:11 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:11.825+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:12.132 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:11 smithi153 podman[71137]: 2024-03-16 12:26:11.84071504 +0000 UTC m=+5.571888963 container died 62124243dba87b907647c024f3c236512988362ca6dd2ce86c603ddbd3faba9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-16T12:26:12.466 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:12 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:12.213+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:12.466 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:12 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:12.213+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:12.466 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:12 smithi153 podman[71137]: 2024-03-16 12:26:12.252802614 +0000 UTC m=+5.983976520 container cleanup 62124243dba87b907647c024f3c236512988362ca6dd2ce86c603ddbd3faba9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:26:12.466 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:12 smithi153 bash[71137]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4 2024-03-16T12:26:12.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:12 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:12.247+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:12.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:12 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:12.247+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:12.567 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:12 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:12.123+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:12.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:12 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:12.286+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:12.751 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:12 smithi153 podman[71154]: 2024-03-16 12:26:12.487563478 +0000 UTC m=+0.645303810 container remove 62124243dba87b907647c024f3c236512988362ca6dd2ce86c603ddbd3faba9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:26:12.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:12 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:12.510+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:12.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:12 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:12.572+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:13.153 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:12 smithi153 podman[71245]: 2024-03-16 12:26:12.822001964 +0000 UTC m=+0.017558420 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:13.153 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:12 smithi153 podman[71245]: 2024-03-16 12:26:12.938513776 +0000 UTC m=+0.134070229 container create 3eb3e872f98121979d09056aaf37aa7dd6016c3560a330e005a19e5b8c8b0282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:13.153 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:13 smithi153 podman[71245]: 2024-03-16 12:26:13.158630829 +0000 UTC m=+0.354187291 container init 3eb3e872f98121979d09056aaf37aa7dd6016c3560a330e005a19e5b8c8b0282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:26:13.153 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:13 smithi153 podman[71245]: 2024-03-16 12:26:13.161752217 +0000 UTC m=+0.357308670 container start 3eb3e872f98121979d09056aaf37aa7dd6016c3560a330e005a19e5b8c8b0282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:13.304 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:13 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:13.081+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:13.304 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:12 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:12.873+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:13.425 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:13 smithi153 podman[71245]: 2024-03-16 12:26:13.257870796 +0000 UTC m=+0.453427264 container attach 3eb3e872f98121979d09056aaf37aa7dd6016c3560a330e005a19e5b8c8b0282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T12:26:13.425 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:13 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:13.172+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:13.425 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:13 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:13.172+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:13.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:13 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:13.258+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:13.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:13 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:13.258+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:13.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:13 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:13.326+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:13.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:13 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:13.501+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:13.751 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 16 12:26:13 smithi153 podman[71245]: 2024-03-16 12:26:13.446956857 +0000 UTC m=+0.642513311 container died 3eb3e872f98121979d09056aaf37aa7dd6016c3560a330e005a19e5b8c8b0282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-16T12:26:13.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:13 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:13.526+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:14.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T12:26:14.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:14.188 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T12:26:14.188 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T12:26:14.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.5 2024-03-16T12:26:14.218 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:13 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:13.912+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:14.218 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:14.097+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:14.434 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:14 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:14.183+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:14.435 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:14 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:14.183+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:14.488 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:14 smithi173 systemd[1]: Stopping Ceph osd.5 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:14.488 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:14.365+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:14.488 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:14.255+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:14.488 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:14.255+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:14.740 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:14 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:14.509+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:14.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:14 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:14.455+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:14.817 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:14.508+0000 7f6032b6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:14.818 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:14.508+0000 7f6032b6d700 -1 osd.5 95 *** Got signal Terminated *** 2024-03-16T12:26:14.818 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:14.508+0000 7f6032b6d700 -1 osd.5 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:15.318 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:15 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:15.144+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:15.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:14.927+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:15.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:14 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:14.927+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:15.483 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:15.202+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:15.483 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:15.202+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:15.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:15.202+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:15.740 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:15 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:15.496+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:15.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:15 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:15.295+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:15.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:15 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:15.295+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:15.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:15 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:15.503+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:15.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:15 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:15.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:16.155 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:15 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:15.913+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:16.156 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:15 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:15.913+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:16.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:16.225+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:16.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:16.225+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:16.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:16.225+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:16.515 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:16.278+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:16.515 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:16.278+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:16.515 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:16.278+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:16.567 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:16.177+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:16.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:16.337+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:16.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:16.535+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:16.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:16 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:16.535+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:17.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:16.528+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:17.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:16 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:16.528+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:17.153 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:16.907+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:17.153 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:16 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:16.907+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:17.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:17 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:17.231+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:17.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:17 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:17.231+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:17.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:17 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:17.231+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:17.542 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:17.292+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:17.542 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:17.292+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:17.542 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:17.292+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:17.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:17 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:17.174+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:17.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:17 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:17.322+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:17.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:17 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:17.482+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:17.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:17 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:17.482+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:17.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:17.561+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:17.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:17.561+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:18.197 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:17 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:17.886+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:18.197 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:17 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:17.886+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:18.478 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:18 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:18.247+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:18.478 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:18 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:18.247+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:18.479 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:18 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:18.247+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:18.567 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:18.218+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:18.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:18.218+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.672798+0000 front 2024-03-16T12:25:52.672825+0000 (oldest deadline 2024-03-16T12:26:17.873511+0000) 2024-03-16T12:26:18.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:18.317+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:18.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:18.317+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:18.584 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:18 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:18.290+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:18.584 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:18 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:18.290+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:18.584 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:18 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:18.290+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:18.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:18 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:18.499+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:18.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:18 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:18.499+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:18.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:18 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:18.604+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:18.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:18 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:18.604+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:19.148 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:18.890+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:19.149 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:18 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:18.890+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:19.501 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:19 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:19.503+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:19.502 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:19 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:19.503+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:19.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:19 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:19.262+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:19.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:19 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:19.262+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:19.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:19 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:19.262+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:19.567 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:19.170+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:44.471912+0000 front 2024-03-16T12:25:44.471967+0000 (oldest deadline 2024-03-16T12:26:07.372002+0000) 2024-03-16T12:26:19.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5[48285]: 2024-03-16T12:26:19.170+0000 7f602b159700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.672798+0000 front 2024-03-16T12:25:52.672825+0000 (oldest deadline 2024-03-16T12:26:17.873511+0000) 2024-03-16T12:26:19.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:19.326+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:19.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:19.326+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:19.585 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:19 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:19.280+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:19.585 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:19 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:19.280+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:19.585 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:19 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:19.280+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:19.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:19 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:19.605+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:19.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:19 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:19.605+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:20.263 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:20 smithi173 podman[71085]: 2024-03-16 12:26:20.037267953 +0000 UTC m=+5.639344586 container died fd64c0deaeae4896b2603ebda8443bf39b62b4e450e19fe88fe81d8662000b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:20.263 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:19.897+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:20.263 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:19 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:19.897+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:20.552 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:20 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:20.273+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:20.552 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:20 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:20.273+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:20.552 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:20 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:20.273+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:20.567 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:20 smithi173 podman[71085]: 2024-03-16 12:26:20.384161404 +0000 UTC m=+5.986238028 container cleanup fd64c0deaeae4896b2603ebda8443bf39b62b4e450e19fe88fe81d8662000b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-16T12:26:20.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:20 smithi173 bash[71085]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5 2024-03-16T12:26:20.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:20 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:20.292+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:20.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:20 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:20.292+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:20.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:20.483+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:20.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:20.483+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:20.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:20.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:20.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:20.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:20.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:20 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:20.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:20.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:20 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:20.572+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:20.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:20 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:20.572+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:21.021 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:20 smithi173 podman[71102]: 2024-03-16 12:26:20.702421668 +0000 UTC m=+0.668334655 container remove fd64c0deaeae4896b2603ebda8443bf39b62b4e450e19fe88fe81d8662000b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:21.022 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:20 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:20.894+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:21.022 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:20 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:20.894+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:21.293 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:20.946529033 +0000 UTC m=+0.018023436 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:21.293 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:21.070168595 +0000 UTC m=+0.141662980 container create 99fcdd179a8f10c14eb94594d59bc10118fde9da2674a4aff8de17bb5d2310fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:21.293 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:21 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:21.264+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:21.294 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:21 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:21.264+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:21.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:21.225+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:21.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:21.225+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:21.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:21.225+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:21.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:21.314937645 +0000 UTC m=+0.386432041 container init 99fcdd179a8f10c14eb94594d59bc10118fde9da2674a4aff8de17bb5d2310fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:26:21.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:21.318268645 +0000 UTC m=+0.389763040 container start 99fcdd179a8f10c14eb94594d59bc10118fde9da2674a4aff8de17bb5d2310fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:26:21.568 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:21.406722893 +0000 UTC m=+0.478217313 container attach 99fcdd179a8f10c14eb94594d59bc10118fde9da2674a4aff8de17bb5d2310fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T12:26:21.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:21 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:21.471+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:21.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:21 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:21.471+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:21.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:21 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:21.356+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:21.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:21 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:21.356+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:21.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:21 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:21.356+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:21.865 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 16 12:26:21 smithi173 podman[71190]: 2024-03-16 12:26:21.6065485 +0000 UTC m=+0.678042898 container died 99fcdd179a8f10c14eb94594d59bc10118fde9da2674a4aff8de17bb5d2310fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T12:26:21.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:21.617+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:21.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:21 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:21.617+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:22.205 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:21 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:21.887+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:22.205 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:21 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:21.887+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:22.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T12:26:22.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:22.409 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T12:26:22.409 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T12:26:22.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.6 2024-03-16T12:26:22.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:22.194+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:22.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:22.194+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:22.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:22.194+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:22.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:22 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:22.226+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:22.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:22 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:22.226+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:22.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:22 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:22.435+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:22.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:22 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:22.435+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:22.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:22 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:22.347+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:22.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:22 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:22.347+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:22.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:22 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:22.347+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:22.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:22 smithi098 systemd[1]: Stopping Ceph osd.6 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:22.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:22.630+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:22.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:22.630+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:23.213 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:22 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:22.904+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:23.214 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:22 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:22.904+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:23.490 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:23.027+0000 7fdb86800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:23.027+0000 7fdb86800700 -1 osd.6 95 *** Got signal Terminated *** 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:23.027+0000 7fdb86800700 -1 osd.6 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:23.175+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:23.175+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:23.175+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:23.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:23.175+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:23.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:23 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:23.235+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:23.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:23 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:23.235+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:23.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:23.468+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:23.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:23.468+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:23.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:23.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:23.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:23.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:23.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:23 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:23.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:23.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:23.675+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:23.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:23 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:23.675+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:24.177 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:23 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:23.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:24.177 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:23 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:23.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:24.177 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:24.198+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:24.177 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:24.198+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:24.178 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:24.198+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:24.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:24.160+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:24.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:24.160+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:24.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:24.160+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:24.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:24.160+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:24.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:24.474+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:24.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:24.474+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:24.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:24.352+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:24.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:24.352+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:24.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:24 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:24.352+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:24.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:24.720+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:24.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:24.720+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:24.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:24 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:24.720+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:58.118628+0000 front 2024-03-16T12:25:58.119101+0000 (oldest deadline 2024-03-16T12:26:24.018479+0000) 2024-03-16T12:26:25.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:24.838+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:25.068 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:24 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:24.838+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:25.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:25.185+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:25.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:25.185+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:25.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:25.185+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:25.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:25.185+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:25.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:25.191+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:25.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:25.191+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:25.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:25.191+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:25.470+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:25.470+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:25.470+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:25.400+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:25.400+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:25.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:25 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:25.400+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:25.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:25.670+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:25.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:25.670+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:25.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:25 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:25.670+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:58.118628+0000 front 2024-03-16T12:25:58.119101+0000 (oldest deadline 2024-03-16T12:26:24.018479+0000) 2024-03-16T12:26:26.203 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:25.888+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:26.203 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:25 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:25.888+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:26.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:26.214+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:26.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:26.214+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:26.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:26.214+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:26.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:26.214+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:26.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:26 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:26.224+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:26.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:26 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:26.224+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:26.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:26 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:26.224+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:26.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:26.510+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:26.510+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:26.510+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:26.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:26.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:26.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:26.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:26 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:26.364+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:26.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:26.641+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:26.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:26.641+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:26.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:26 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:26.641+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:58.118628+0000 front 2024-03-16T12:25:58.119101+0000 (oldest deadline 2024-03-16T12:26:24.018479+0000) 2024-03-16T12:26:27.246 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:26 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:26.919+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:27.246 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:26 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:26.919+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:27.365 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:27.187+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:27.366 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:27.187+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:27.366 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:27.187+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:27.366 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:27.187+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:27.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:27.267+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:27.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:27.267+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:27.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:27.267+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:27.502+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:27.502+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:27.502+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:27.316+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:27.316+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:27.316+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:27.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:27 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:27.316+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:27.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:27.651+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:43.515190+0000 front 2024-03-16T12:25:43.515105+0000 (oldest deadline 2024-03-16T12:26:07.014889+0000) 2024-03-16T12:26:27.990 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:27.651+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.116815+0000 front 2024-03-16T12:25:51.117409+0000 (oldest deadline 2024-03-16T12:26:16.416772+0000) 2024-03-16T12:26:27.991 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:27 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6[59886]: 2024-03-16T12:26:27.651+0000 7fdb7d000700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:58.118628+0000 front 2024-03-16T12:25:58.119101+0000 (oldest deadline 2024-03-16T12:26:24.018479+0000) 2024-03-16T12:26:28.206 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:27.938+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:28.206 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:27.938+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:28.206 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:27 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:27.938+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:28.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:28.227+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:28.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:28.227+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:28.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:28.227+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:28.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:28.227+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:28.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:28.227+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:28.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:28.227+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:28.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:28.227+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:28.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:28.467+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:28.467+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:28.467+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:28.289+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:28.289+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:28.289+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:28.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:28 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:28.289+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:28.840 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:28 smithi098 podman[92216]: 2024-03-16 12:26:28.527142733 +0000 UTC m=+5.648880765 container died ba51d04baec70ba7cc112441431b77cc983c2c6742c989092c07bf1ad8c756a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T12:26:29.191 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:29 smithi098 podman[92216]: 2024-03-16 12:26:29.080669136 +0000 UTC m=+6.202407194 container cleanup ba51d04baec70ba7cc112441431b77cc983c2c6742c989092c07bf1ad8c756a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T12:26:29.191 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:29 smithi098 bash[92216]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6 2024-03-16T12:26:29.233 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:28.943+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:29.233 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:28.943+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:29.233 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:28 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:28.943+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:29.490 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:29 smithi098 podman[92245]: 2024-03-16 12:26:29.315515857 +0000 UTC m=+0.780189173 container remove ba51d04baec70ba7cc112441431b77cc983c2c6742c989092c07bf1ad8c756a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:29.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:29 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:29.210+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:29.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:29 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:29.210+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:29.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:29 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:29.210+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:29.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:29 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:29.210+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:29.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:29.254+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:29.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:29.254+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:29.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:29.254+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:29.427+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:29.427+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:29.427+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:29.304+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:29.304+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:29.304+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:29.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:29 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:29.304+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:30.317 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:30.230+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:30.318 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:30.230+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:30.318 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:30.230+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:30.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:29.992+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:30.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:29.992+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:30.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:29 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:29.992+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:30.391 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:30 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:30.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:30.391 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:30 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:30.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:30.391 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:30 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:30.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:30.391 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:30 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:30.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:30.391 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.037547783 +0000 UTC m=+0.034324998 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:30.392 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.141486635 +0000 UTC m=+0.138263887 container create 9410aa14328ffb150a24545fea173debec3f4b8385633045d13cc48dbcb39900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-16T12:26:30.392 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.412047993 +0000 UTC m=+0.408825252 container init 9410aa14328ffb150a24545fea173debec3f4b8385633045d13cc48dbcb39900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-16T12:26:30.740 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.423990982 +0000 UTC m=+0.420768226 container start 9410aa14328ffb150a24545fea173debec3f4b8385633045d13cc48dbcb39900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-16T12:26:30.740 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.513012762 +0000 UTC m=+0.509790031 container attach 9410aa14328ffb150a24545fea173debec3f4b8385633045d13cc48dbcb39900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-16T12:26:30.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:30.440+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:30.440+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:30.440+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:30.335+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:30.335+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:30.335+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:30.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:30 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:30.335+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:31.199 INFO:journalctl@ceph.osd.6.smithi098.stdout:Mar 16 12:26:30 smithi098 podman[92336]: 2024-03-16 12:26:30.852290213 +0000 UTC m=+0.849067472 container died 9410aa14328ffb150a24545fea173debec3f4b8385633045d13cc48dbcb39900 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-16T12:26:31.241 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:30.951+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:31.241 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:30.951+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:31.241 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:30 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:30.951+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:31.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:31.246+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:31.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:31.246+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:31.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:31.246+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:31.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:31 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:31.246+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:31.513 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:31.261+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:31.514 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:31.261+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:31.514 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:31.261+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:31.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T12:26:31.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:31.704 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T12:26:31.704 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T12:26:31.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.7 2024-03-16T12:26:31.733 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:31.458+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:31.733 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:31.458+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:31.733 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:31.458+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:31.733 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:31.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:31.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:31.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:31.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:31.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:31.734 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:31 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:31.310+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:32.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:31 smithi153 systemd[1]: Stopping Ceph osd.7 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:32.275 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:31.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:32.275 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:31.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:32.275 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:31 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:31.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:32.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.038+0000 7fd2d3399700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:32.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.038+0000 7fd2d3399700 -1 osd.7 95 *** Got signal Terminated *** 2024-03-16T12:26:32.484 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.038+0000 7fd2d3399700 -1 osd.7 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:32.484 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:32.286+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:32.286+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:32.286+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:32.485 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:32.286+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:32.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:32 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:32.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:32.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:32 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:32.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:32.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:32 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:32.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:32.491 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:32 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:32.233+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:32.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:32.296+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:32.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:32.296+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:32.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:32.296+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:32.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:32.296+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:32.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.505+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:32.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.505+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:32.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:32 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:32.505+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:33.273 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:32.872+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:33.273 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:32.872+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:33.274 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:32 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:32.872+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:33.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:33.248+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:33.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:33.248+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:33.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:33.248+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:33.490 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:33 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:33.248+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:33.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:33.294+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:33.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:33.294+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:33.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:33.294+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:33.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:33.294+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:33.534+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:33.534+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:33.534+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:33.534+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.118968+0000 front 2024-03-16T12:26:09.119008+0000 (oldest deadline 2024-03-16T12:26:32.619106+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:33.288+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:33.288+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:33.288+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:33.288+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:33.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:33.288+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:34.317 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:33.860+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:34.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:33.860+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:34.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:33 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:33.860+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:34.530 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:34.285+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:34.530 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:34.285+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:34.530 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:34.285+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:34.530 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:34.285+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:34.530 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:34.285+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:34.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:34.338+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:34.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:34.338+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:34.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:34.338+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:34.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:34.338+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:34.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:34.293+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:34.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:34.293+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:34.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:34.293+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:34.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:34.293+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:34.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:34 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:34.293+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:35.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:34.550+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:35.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:34.550+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:35.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:34.550+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:35.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:34 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:34.550+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.118968+0000 front 2024-03-16T12:26:09.119008+0000 (oldest deadline 2024-03-16T12:26:32.619106+0000) 2024-03-16T12:26:35.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:34.877+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:35.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:34.877+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:35.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:34 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:34.877+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:35.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:35.268+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:35.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:35.268+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:35.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:35.268+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:35.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:35.268+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:35.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:35.268+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:35.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:35.330+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:35.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:35.330+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:35.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:35.330+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:35.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:35.330+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:35.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:35 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:35.330+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:35.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:35.384+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:35.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:35.384+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:35.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:35.384+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:35.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:35.384+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:36.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:35.568+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:36.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:35.568+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:36.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:35.568+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:36.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:35 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:35.569+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.118968+0000 front 2024-03-16T12:26:09.119008+0000 (oldest deadline 2024-03-16T12:26:32.619106+0000) 2024-03-16T12:26:36.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:35.869+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:36.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:35.869+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:36.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:35.869+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:36.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:35 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:35.869+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:36.549 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:36.291+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:36.549 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:36.291+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:36.549 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:36.291+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:36.549 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:36.291+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:36.550 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:36.291+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:36.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:36.332+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:36.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:36.332+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:36.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:36.332+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:36.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:36.332+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:36.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:36 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:36.332+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:36.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:36.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:36.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:36.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:36.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:36.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:36.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:36.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:37.001 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:36.570+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.215315+0000 front 2024-03-16T12:25:42.215180+0000 (oldest deadline 2024-03-16T12:26:07.515326+0000) 2024-03-16T12:26:37.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:36.570+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.917073+0000 front 2024-03-16T12:25:53.916937+0000 (oldest deadline 2024-03-16T12:26:16.216987+0000) 2024-03-16T12:26:37.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:36.570+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:59.118452+0000 front 2024-03-16T12:25:59.118467+0000 (oldest deadline 2024-03-16T12:26:25.018206+0000) 2024-03-16T12:26:37.002 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:36 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7[55319]: 2024-03-16T12:26:36.570+0000 7fd2cf98d700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.118968+0000 front 2024-03-16T12:26:09.119008+0000 (oldest deadline 2024-03-16T12:26:32.619106+0000) 2024-03-16T12:26:37.317 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:36.857+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:37.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:36.857+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:37.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:36.857+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:37.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:36 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:36.857+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:37.501 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:37 smithi153 podman[71350]: 2024-03-16 12:26:37.40239566 +0000 UTC m=+5.486056247 container died 52b2c283214eb9c6bd6787fc12766972a3d8e5563fa8983b3e3756b2af5675d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T12:26:37.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:37.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:37.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:37.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:37.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:37.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:37.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:37.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:37.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:37.250+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:37.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:37.324+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:37.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:37.324+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:37.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:37.324+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:37.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:37.324+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:37.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:37 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:37.324+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:37.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:37.374+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:37.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:37.374+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:37.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:37.374+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:37.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:37.374+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:38.044 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:37 smithi153 podman[71350]: 2024-03-16 12:26:37.813790361 +0000 UTC m=+5.897450920 container cleanup 52b2c283214eb9c6bd6787fc12766972a3d8e5563fa8983b3e3756b2af5675d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-16T12:26:38.044 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:37 smithi153 bash[71350]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7 2024-03-16T12:26:38.317 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:37.882+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:38.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:37.882+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:38.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:37.882+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:38.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:37 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:37.882+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71368]: 2024-03-16 12:26:38.065449624 +0000 UTC m=+0.664065517 container remove 52b2c283214eb9c6bd6787fc12766972a3d8e5563fa8983b3e3756b2af5675d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:38 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:38.243+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:38 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:38.243+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:38 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:38.243+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:38 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:38.243+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:38.453 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:38 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:38.243+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:38.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:38.373+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:38.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:38.373+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:38.704 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:38.373+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:38.704 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:38.373+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:38.704 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:38 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:38.373+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:38.751 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71457]: 2024-03-16 12:26:38.474593702 +0000 UTC m=+0.110521493 container create 158d64b7e405c745c865e9cd30b22813fd4ba5df6611c1bbf6c124c66eae79a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T12:26:38.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71457]: 2024-03-16 12:26:38.381819254 +0000 UTC m=+0.017747055 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:38.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71457]: 2024-03-16 12:26:38.719168555 +0000 UTC m=+0.355096359 container init 158d64b7e405c745c865e9cd30b22813fd4ba5df6611c1bbf6c124c66eae79a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T12:26:38.752 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71457]: 2024-03-16 12:26:38.722519457 +0000 UTC m=+0.358447250 container start 158d64b7e405c745c865e9cd30b22813fd4ba5df6611c1bbf6c124c66eae79a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:38.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:38.406+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:38.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:38.406+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:38.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:38.406+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:38.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:38.406+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:39.008 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:38 smithi153 podman[71457]: 2024-03-16 12:26:38.80194829 +0000 UTC m=+0.437876090 container attach 158d64b7e405c745c865e9cd30b22813fd4ba5df6611c1bbf6c124c66eae79a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T12:26:39.008 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 16 12:26:39 smithi153 podman[71457]: 2024-03-16 12:26:39.012604618 +0000 UTC m=+0.648532411 container died 158d64b7e405c745c865e9cd30b22813fd4ba5df6611c1bbf6c124c66eae79a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T12:26:39.262 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:39 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:39.260+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:39.262 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:39 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:39.260+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:39.262 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:39 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:39.260+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:39.263 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:39 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:39.260+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:39.263 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:39 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:39.260+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:39.317 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:38.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:39.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:38.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:39.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:38.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:39.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:38 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:38.878+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:39.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T12:26:39.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:39.685 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T12:26:39.685 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T12:26:39.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.8 2024-03-16T12:26:39.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:39.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:39.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:39.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:39.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:39.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:39.715 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:39.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:39.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:39.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:39.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:39.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:39.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:39.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:39.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:39.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:39.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:39 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:39.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:39.992 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:39 smithi173 systemd[1]: Stopping Ceph osd.8 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:39.992 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:39.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:39.992 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:39.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:39.992 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:39.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:39.993 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:39 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:39.902+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:39.993 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.012+0000 7fc1bde0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:39.993 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.012+0000 7fc1bde0e700 -1 osd.8 95 *** Got signal Terminated *** 2024-03-16T12:26:39.993 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.012+0000 7fc1bde0e700 -1 osd.8 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:40.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:40 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:40.238+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:40.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:40 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:40.238+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:40.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:40 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:40.238+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:40.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:40 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:40.238+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:40.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:40 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:40.238+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:40.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:40.409+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:40.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:40.409+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:40.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:40.409+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:40.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:40.409+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:40.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:40 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:40.409+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:40.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:40.410+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:40.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:40.410+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:40.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:40.410+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:40.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:40.410+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:41.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.932+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:41.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.932+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:41.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.932+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:41.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.932+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:41.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:40 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:40.932+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:16.016951+0000 front 2024-03-16T12:26:16.016893+0000 (oldest deadline 2024-03-16T12:26:40.617442+0000) 2024-03-16T12:26:41.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:41.204+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:41.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:41.204+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:41.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:41.204+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:41.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:41.204+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:41.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:41.204+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:41.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:41.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:41.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:41.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:41.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:41.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:41 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:41.431+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:41.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:41.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:41.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:41.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:41.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:41.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:41.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:41.397+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:42.317 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:41.968+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:42.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:41.968+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:42.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:41.968+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:42.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:41.968+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:42.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:41 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:41.968+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:16.016951+0000 front 2024-03-16T12:26:16.016893+0000 (oldest deadline 2024-03-16T12:26:40.617442+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.248+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.248+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.248+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.249+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.249+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:42.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:42 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:42.249+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:42.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:42.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:42.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:42.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:42.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:42.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:42 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:42.420+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:42.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:42.371+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:42.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:42.371+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:42.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:42.371+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:42.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:42.371+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:43.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:42.931+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:43.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:42.931+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:43.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:42.931+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:43.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:42.931+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:43.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:42 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:42.931+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:16.016951+0000 front 2024-03-16T12:26:16.016893+0000 (oldest deadline 2024-03-16T12:26:40.617442+0000) 2024-03-16T12:26:43.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:43.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:43.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:43.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:43.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:43.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:43 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:43.201+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:43.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:43.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:43.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:43.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:43.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:43.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:43 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:43.398+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:43.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:43.356+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:43.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:43.356+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:43.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:43.356+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:43.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:43.356+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:44.315 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:43.916+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:44.316 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:43.916+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:44.316 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:43.916+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:44.316 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:43.916+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:44.316 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:43 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:43.916+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:16.016951+0000 front 2024-03-16T12:26:16.016893+0000 (oldest deadline 2024-03-16T12:26:40.617442+0000) 2024-03-16T12:26:44.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:44.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:44.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:44.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:44.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:44.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:44 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:44.189+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:44.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:44.336+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:44.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:44.336+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:44.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:44.336+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:44.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:44.336+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:44.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:44.336+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:44.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:44.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:44.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:44.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:44.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:44.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:44 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:44.395+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:45.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:44.906+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:41.309960+0000 front 2024-03-16T12:25:41.309849+0000 (oldest deadline 2024-03-16T12:26:06.610042+0000) 2024-03-16T12:26:45.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:44.906+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.312201+0000 front 2024-03-16T12:25:52.312162+0000 (oldest deadline 2024-03-16T12:26:14.612287+0000) 2024-03-16T12:26:45.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:44.906+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:02.114532+0000 front 2024-03-16T12:26:02.114697+0000 (oldest deadline 2024-03-16T12:26:27.914783+0000) 2024-03-16T12:26:45.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:44.906+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.616003+0000 front 2024-03-16T12:26:09.616119+0000 (oldest deadline 2024-03-16T12:26:34.915920+0000) 2024-03-16T12:26:45.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:44 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8[55285]: 2024-03-16T12:26:44.906+0000 7fc1ba402700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:16.016951+0000 front 2024-03-16T12:26:16.016893+0000 (oldest deadline 2024-03-16T12:26:40.617442+0000) 2024-03-16T12:26:45.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:45.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:45.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:45.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:45.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:45.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:45.228+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:45.709 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:45 smithi173 podman[71306]: 2024-03-16 12:26:45.500563666 +0000 UTC m=+5.595415456 container died d47cc4fc6391ebf382f36e2e47fe172f003fe3c2853c206b4aa6ff32d1529588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:45.709 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:45 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:45.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:45.709 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:45 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:45.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:45.709 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:45 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:45.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:45.710 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:45 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:45.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:45.710 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:45 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:45.378+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:45.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:45.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:45.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:45.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:45.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:45.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:45 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:45.388+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:46.026 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:45 smithi173 podman[71306]: 2024-03-16 12:26:45.838471473 +0000 UTC m=+5.933323195 container cleanup d47cc4fc6391ebf382f36e2e47fe172f003fe3c2853c206b4aa6ff32d1529588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T12:26:46.026 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:45 smithi173 bash[71306]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8 2024-03-16T12:26:46.318 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71320]: 2024-03-16 12:26:46.048190804 +0000 UTC m=+0.542101053 container remove d47cc4fc6391ebf382f36e2e47fe172f003fe3c2853c206b4aa6ff32d1529588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T12:26:46.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:46.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:46.501 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:46.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:46.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:46.502 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:46 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:46.242+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:46.586 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71417]: 2024-03-16 12:26:46.351551284 +0000 UTC m=+0.018237991 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71417]: 2024-03-16 12:26:46.490910632 +0000 UTC m=+0.157597324 container create 7f4fc81c0a75bb1ebfebc59e2537a1eef9aaba8bbaf3f56dc696161e1842068b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:46 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:46.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:46 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:46.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:46 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:46.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:46 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:46.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:46.587 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:46 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:46.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:46.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:46.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:46.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:46.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:46.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:46.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:46 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:46.348+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:47.017 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71417]: 2024-03-16 12:26:46.752225358 +0000 UTC m=+0.418912054 container init 7f4fc81c0a75bb1ebfebc59e2537a1eef9aaba8bbaf3f56dc696161e1842068b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-03-16T12:26:47.017 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71417]: 2024-03-16 12:26:46.755693136 +0000 UTC m=+0.422379830 container start 7f4fc81c0a75bb1ebfebc59e2537a1eef9aaba8bbaf3f56dc696161e1842068b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T12:26:47.017 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:46 smithi173 podman[71417]: 2024-03-16 12:26:46.83543997 +0000 UTC m=+0.502126677 container attach 7f4fc81c0a75bb1ebfebc59e2537a1eef9aaba8bbaf3f56dc696161e1842068b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T12:26:47.286 INFO:journalctl@ceph.osd.8.smithi173.stdout:Mar 16 12:26:47 smithi173 podman[71417]: 2024-03-16 12:26:47.03924644 +0000 UTC m=+0.705933149 container died 7f4fc81c0a75bb1ebfebc59e2537a1eef9aaba8bbaf3f56dc696161e1842068b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T12:26:47.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:47 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:47.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:47.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:47 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:47.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:47.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:47 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:47.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:47.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:47 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:47.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:47.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:47 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:47.340+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:47.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T12:26:47.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:47.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:47.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:47.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:47.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:47.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:47.333+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:47.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:47.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:47.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:47.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:47.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:47.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:47 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:47.275+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:47.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:47.781 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T12:26:47.781 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T12:26:47.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.9 2024-03-16T12:26:48.240 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:47 smithi098 systemd[1]: Stopping Ceph osd.9 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:48.261 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:48.261 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:48.261 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:48.261 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:48.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:48.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:48.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:48.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:48.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:48.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:48.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:48.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:48.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:48 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:48.348+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:48.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.334+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.366+0000 7f21ca800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.366+0000 7f21ca800700 -1 osd.9 95 *** Got signal Terminated *** 2024-03-16T12:26:48.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:48 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:48.366+0000 7f21ca800700 -1 osd.9 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:48.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:48.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:48 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:48.280+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:49.703 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:49.703 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:49.703 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:49.703 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:49.703 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:49.704 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:49.704 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:49 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:49.330+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:49.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:49.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:49.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:49.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:49.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:49.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:49 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:49.326+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:49.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:49 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:49.376+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:50.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:50.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:50.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:50.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:50.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:50.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:50 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:50.279+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:50.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:50.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:50 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:50.372+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:50.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:50.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:50.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:50.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:50.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:50.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:50.424+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:51.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:51.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:51.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:51.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:51.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:51.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:51 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:51.294+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:51.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:51.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:51.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:51.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:51.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:51.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:51.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:51 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:51.385+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:51.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:51.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:51.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:51.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:51.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:51.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:51 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:51.448+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:52.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:52.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:52.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:52.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:52.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:52.741 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:52 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:52.302+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:52.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:52 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:52.414+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:52.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:52.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:52.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:52.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:52.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:52.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:52 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:52.442+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:53.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:33.657023+0000 front 2024-03-16T12:25:33.657716+0000 (oldest deadline 2024-03-16T12:25:57.757256+0000) 2024-03-16T12:26:53.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:45.360171+0000 front 2024-03-16T12:25:45.360362+0000 (oldest deadline 2024-03-16T12:26:11.260391+0000) 2024-03-16T12:26:53.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:51.261252+0000 front 2024-03-16T12:25:51.261545+0000 (oldest deadline 2024-03-16T12:26:15.961325+0000) 2024-03-16T12:26:53.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.663552+0000 front 2024-03-16T12:26:00.663444+0000 (oldest deadline 2024-03-16T12:26:22.963095+0000) 2024-03-16T12:26:53.703 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:07.064920+0000 front 2024-03-16T12:26:07.065017+0000 (oldest deadline 2024-03-16T12:26:34.065856+0000) 2024-03-16T12:26:53.704 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9[66956]: 2024-03-16T12:26:53.337+0000 7f21c6000700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:15.767525+0000 front 2024-03-16T12:26:15.767581+0000 (oldest deadline 2024-03-16T12:26:40.467784+0000) 2024-03-16T12:26:53.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:53.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:53.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:53.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:53.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:53.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:53.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:53 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:53.448+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:53.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:53.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:53.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:53.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:53.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:53.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:53.480+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:53.990 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:53 smithi098 podman[92442]: 2024-03-16 12:26:53.724899368 +0000 UTC m=+5.504770370 container died 1075a1b07297fa6143d35a17e6e74edfb70a3c6e5f625b5464af0cfac464f92c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:26:54.365 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:54 smithi098 podman[92442]: 2024-03-16 12:26:54.135814718 +0000 UTC m=+5.915685763 container cleanup 1075a1b07297fa6143d35a17e6e74edfb70a3c6e5f625b5464af0cfac464f92c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:54.366 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:54 smithi098 bash[92442]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9 2024-03-16T12:26:54.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:54 smithi098 podman[92461]: 2024-03-16 12:26:54.386798946 +0000 UTC m=+0.647903306 container remove 1075a1b07297fa6143d35a17e6e74edfb70a3c6e5f625b5464af0cfac464f92c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:54.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:54.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:54 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:54.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:54.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:54.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:54.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:54.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:54.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:54.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:54 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:54.491+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:55.428 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.015463998 +0000 UTC m=+0.032813899 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:26:55.428 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.171147323 +0000 UTC m=+0.188497261 container create 510805eaa52871da228b348b79781a1e8edd0eff7bc9b13ca7924738346cf4d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T12:26:55.428 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.44952294 +0000 UTC m=+0.466872885 container init 510805eaa52871da228b348b79781a1e8edd0eff7bc9b13ca7924738346cf4d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:26:55.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.461804396 +0000 UTC m=+0.479154306 container start 510805eaa52871da228b348b79781a1e8edd0eff7bc9b13ca7924738346cf4d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:26:55.740 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.540290376 +0000 UTC m=+0.557640337 container attach 510805eaa52871da228b348b79781a1e8edd0eff7bc9b13ca7924738346cf4d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T12:26:55.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:55.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:55 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:55.510+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:55.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:55.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:55.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:55.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:55.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:55.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:55 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:55.465+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:56.231 INFO:journalctl@ceph.osd.9.smithi098.stdout:Mar 16 12:26:55 smithi098 podman[92552]: 2024-03-16 12:26:55.886808493 +0000 UTC m=+0.904158405 container died 510805eaa52871da228b348b79781a1e8edd0eff7bc9b13ca7924738346cf4d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True) 2024-03-16T12:26:56.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T12:26:56.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:26:56.678 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T12:26:56.679 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T12:26:56.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.10 2024-03-16T12:26:56.723 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:56.723 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:56.723 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:56.723 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:56.724 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:56.724 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:56.724 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:56.491+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:56.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:56.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:56.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:56.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:56.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:56.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:56 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:56.487+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:56.986 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:56 smithi153 systemd[1]: Stopping Ceph osd.10 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:26:57.251 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.007+0000 7f4423623700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:26:57.251 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.007+0000 7f4423623700 -1 osd.10 95 *** Got signal Terminated *** 2024-03-16T12:26:57.251 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.007+0000 7f4423623700 -1 osd.10 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:57.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:57.485+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:58.001 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:58.002 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:57 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:57.523+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:58.568 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:58 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:58.490+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:26:58.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:58.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:58.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:58.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:58.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:58.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:58.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:58 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:58.486+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:59.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:26:59.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:26:59.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:26:59.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:26:59.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:26:59.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:26:59.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:26:59 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:26:59.472+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:26:59.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:26:59.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:26:59.819 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:26:59.819 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:26:59.819 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:26:59.819 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:26:59 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:26:59.497+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:00.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:27:00.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:27:00.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:27:00.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:27:00.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:27:00.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:27:00.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:00 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:00.459+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:27:00.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:00.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:00.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:00.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:00.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:00.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:00.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:00 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:00.532+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:01.751 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.286402+0000 front 2024-03-16T12:25:36.286139+0000 (oldest deadline 2024-03-16T12:25:59.186018+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-03-16T12:25:42.587623+0000 front 2024-03-16T12:25:42.587787+0000 (oldest deadline 2024-03-16T12:26:08.487758+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:53.188762+0000 front 2024-03-16T12:25:53.188687+0000 (oldest deadline 2024-03-16T12:26:14.888746+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:26:00.789995+0000 front 2024-03-16T12:26:00.789830+0000 (oldest deadline 2024-03-16T12:26:25.990050+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:09.490954+0000 front 2024-03-16T12:26:09.490973+0000 (oldest deadline 2024-03-16T12:26:32.990948+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:18.292033+0000 front 2024-03-16T12:26:18.292018+0000 (oldest deadline 2024-03-16T12:26:41.791932+0000) 2024-03-16T12:27:01.752 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:01 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10[61970]: 2024-03-16T12:27:01.497+0000 7f441fc17700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:25.594571+0000 front 2024-03-16T12:26:25.594485+0000 (oldest deadline 2024-03-16T12:26:49.094203+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:01.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:01 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:01.535+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:02.603 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:02 smithi153 podman[71576]: 2024-03-16 12:27:02.373806507 +0000 UTC m=+5.485066898 container died a54555d7bd83a1c0f2a7e26d1fb2e191060a2192f7ad82f5288fb87f9a815f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-16T12:27:02.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:02.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:02.549+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:02.869 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:02 smithi153 podman[71576]: 2024-03-16 12:27:02.715567413 +0000 UTC m=+5.826827792 container cleanup a54555d7bd83a1c0f2a7e26d1fb2e191060a2192f7ad82f5288fb87f9a815f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T12:27:02.869 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:02 smithi153 bash[71576]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10 2024-03-16T12:27:03.251 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:02 smithi153 podman[71593]: 2024-03-16 12:27:02.891571045 +0000 UTC m=+0.513898131 container remove a54555d7bd83a1c0f2a7e26d1fb2e191060a2192f7ad82f5288fb87f9a815f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-16T12:27:03.607 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.325822112 +0000 UTC m=+0.113458484 container create 9c50c05832d4e7ce0c6ad9e0bf438b93d95c24616ce71d2446f248dcc7d42b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-16T12:27:03.607 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.23038909 +0000 UTC m=+0.018025421 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:27:03.607 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.561557894 +0000 UTC m=+0.349194216 container init 9c50c05832d4e7ce0c6ad9e0bf438b93d95c24616ce71d2446f248dcc7d42b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T12:27:03.607 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.564767986 +0000 UTC m=+0.352404318 container start 9c50c05832d4e7ce0c6ad9e0bf438b93d95c24616ce71d2446f248dcc7d42b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:27:03.607 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.628156655 +0000 UTC m=+0.415792998 container attach 9c50c05832d4e7ce0c6ad9e0bf438b93d95c24616ce71d2446f248dcc7d42b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:27:03.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:03.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:03.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:03.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:03.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:03.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:03.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:03 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:03.551+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:04.001 INFO:journalctl@ceph.osd.10.smithi153.stdout:Mar 16 12:27:03 smithi153 podman[71683]: 2024-03-16 12:27:03.849659196 +0000 UTC m=+0.637295519 container died 9c50c05832d4e7ce0c6ad9e0bf438b93d95c24616ce71d2446f248dcc7d42b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T12:27:04.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T12:27:04.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:04.459 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T12:27:04.459 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T12:27:04.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.11 2024-03-16T12:27:04.788 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 systemd[1]: Stopping Ceph osd.11 for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:04.789 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.545+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:05.067 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.809+0000 7f1422075700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:05.067 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.809+0000 7f1422075700 -1 osd.11 95 *** Got signal Terminated *** 2024-03-16T12:27:05.067 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:04 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:04.809+0000 7f1422075700 -1 osd.11 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:27:05.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:05.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:05 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:05.546+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:06.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:06.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:06 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:06.509+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:07.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:07.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:07 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:07.522+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-03-16T12:26:42.636359+0000 front 2024-03-16T12:26:42.636389+0000 (oldest deadline 2024-03-16T12:27:07.336389+0000) 2024-03-16T12:27:08.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:08.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:08 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:08.560+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-03-16T12:26:42.636359+0000 front 2024-03-16T12:26:42.636389+0000 (oldest deadline 2024-03-16T12:27:07.336389+0000) 2024-03-16T12:27:09.817 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-16T12:25:36.726532+0000 front 2024-03-16T12:25:36.726580+0000 (oldest deadline 2024-03-16T12:25:59.626147+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-03-16T12:25:52.230383+0000 front 2024-03-16T12:25:52.230360+0000 (oldest deadline 2024-03-16T12:26:17.530566+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-03-16T12:25:57.531538+0000 front 2024-03-16T12:25:57.531239+0000 (oldest deadline 2024-03-16T12:26:23.430995+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-03-16T12:26:08.132045+0000 front 2024-03-16T12:26:08.132196+0000 (oldest deadline 2024-03-16T12:26:32.231852+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-03-16T12:26:17.532847+0000 front 2024-03-16T12:26:17.532978+0000 (oldest deadline 2024-03-16T12:26:43.933557+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-03-16T12:26:23.934389+0000 front 2024-03-16T12:26:23.934521+0000 (oldest deadline 2024-03-16T12:26:48.634163+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-03-16T12:26:36.235749+0000 front 2024-03-16T12:26:36.235647+0000 (oldest deadline 2024-03-16T12:26:59.735494+0000) 2024-03-16T12:27:09.818 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:09 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11[62192]: 2024-03-16T12:27:09.554+0000 7f141a661700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-03-16T12:26:42.636359+0000 front 2024-03-16T12:26:42.636389+0000 (oldest deadline 2024-03-16T12:27:07.336389+0000) 2024-03-16T12:27:10.429 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:10 smithi173 podman[71549]: 2024-03-16 12:27:10.153281234 +0000 UTC m=+5.476826951 container died faf99055eb2b76db18ab7caeffb9c0454649ba5cd5465f1f0c452358fd1621bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T12:27:10.429 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:10 smithi173 podman[71549]: 2024-03-16 12:27:10.451344514 +0000 UTC m=+5.774890230 container cleanup faf99055eb2b76db18ab7caeffb9c0454649ba5cd5465f1f0c452358fd1621bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T12:27:10.690 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:10 smithi173 bash[71549]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11 2024-03-16T12:27:10.991 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:10 smithi173 podman[71567]: 2024-03-16 12:27:10.712606759 +0000 UTC m=+0.557131308 container remove faf99055eb2b76db18ab7caeffb9c0454649ba5cd5465f1f0c452358fd1621bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T12:27:11.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:10.915796802 +0000 UTC m=+0.017572952 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:27:11.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:11.013465315 +0000 UTC m=+0.115241446 container create 92d710681ac74ae5887ec3e4b289ef8811168cfdbfa84099baa6a7e8692a3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T12:27:11.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:11.217275149 +0000 UTC m=+0.319051298 container init 92d710681ac74ae5887ec3e4b289ef8811168cfdbfa84099baa6a7e8692a3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:27:11.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:11.220794066 +0000 UTC m=+0.322570204 container start 92d710681ac74ae5887ec3e4b289ef8811168cfdbfa84099baa6a7e8692a3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T12:27:11.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:11.300072487 +0000 UTC m=+0.401848641 container attach 92d710681ac74ae5887ec3e4b289ef8811168cfdbfa84099baa6a7e8692a3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-16T12:27:11.567 INFO:journalctl@ceph.osd.11.smithi173.stdout:Mar 16 12:27:11 smithi173 podman[71657]: 2024-03-16 12:27:11.507150352 +0000 UTC m=+0.608926490 container died 92d710681ac74ae5887ec3e4b289ef8811168cfdbfa84099baa6a7e8692a3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T12:27:12.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T12:27:12.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:12.080 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T12:27:12.080 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T12:27:12.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.a 2024-03-16T12:27:12.489 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:12 smithi098 systemd[1]: Stopping Ceph mds.a for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:12.990 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:12 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a[73644]: 2024-03-16T12:27:12.678+0000 7f2ea6a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:12.990 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:12 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a[73644]: 2024-03-16T12:27:12.678+0000 7f2ea6a00700 -1 mds.a *** got signal Terminated *** 2024-03-16T12:27:15.640 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:15 smithi098 podman[92669]: 2024-03-16 12:27:15.384966572 +0000 UTC m=+2.839916720 container died ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:27:15.976 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:15 smithi098 podman[92669]: 2024-03-16 12:27:15.770572259 +0000 UTC m=+3.225522403 container cleanup ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T12:27:15.976 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:15 smithi098 bash[92669]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a 2024-03-16T12:27:16.240 INFO:journalctl@ceph.mds.a.smithi098.stdout:Mar 16 12:27:15 smithi098 podman[92681]: 2024-03-16 12:27:15.996579598 +0000 UTC m=+0.598595380 container remove ef6b0087c4e3c4ef81043671f27f0ebdae03358c1677a40e8afed7f803c3b93b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T12:27:16.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T12:27:16.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:16.620 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T12:27:16.620 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T12:27:16.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.d 2024-03-16T12:27:16.990 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:16 smithi098 systemd[1]: Stopping Ceph mds.d for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:17.490 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d[72639]: 2024-03-16T12:27:17.220+0000 7f7562e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:17.490 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:17 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d[72639]: 2024-03-16T12:27:17.220+0000 7f7562e00700 -1 mds.d *** got signal Terminated *** 2024-03-16T12:27:20.802 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:20 smithi098 podman[92789]: 2024-03-16 12:27:20.521260128 +0000 UTC m=+3.437305248 container died a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:27:21.179 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:20 smithi098 podman[92789]: 2024-03-16 12:27:20.932793965 +0000 UTC m=+3.848839032 container cleanup a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True) 2024-03-16T12:27:21.179 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:20 smithi098 bash[92789]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d 2024-03-16T12:27:21.489 INFO:journalctl@ceph.mds.d.smithi098.stdout:Mar 16 12:27:21 smithi098 podman[92801]: 2024-03-16 12:27:21.200546067 +0000 UTC m=+0.668363029 container remove a54ee58bae3db2cd536b52f13aeeccc0a6241652bf8264102fcdb61c6118a976 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T12:27:21.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T12:27:21.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:21.769 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T12:27:21.769 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T12:27:21.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.g 2024-03-16T12:27:22.240 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:21 smithi098 systemd[1]: Stopping Ceph mds.g for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:22.740 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g[73277]: 2024-03-16T12:27:22.375+0000 7f3f1fa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:22.740 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:22 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g[73277]: 2024-03-16T12:27:22.375+0000 7f3f1fa00700 -1 mds.g *** got signal Terminated *** 2024-03-16T12:27:26.628 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:26 smithi098 podman[92909]: 2024-03-16 12:27:26.391749983 +0000 UTC m=+4.176442526 container died 7b3568f197c27a07884d85e8a5c192b1258c80e75d5155047f5946af84f42d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T12:27:26.958 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:26 smithi098 podman[92909]: 2024-03-16 12:27:26.794841395 +0000 UTC m=+4.579533958 container cleanup 7b3568f197c27a07884d85e8a5c192b1258c80e75d5155047f5946af84f42d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T12:27:26.958 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:26 smithi098 bash[92909]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g 2024-03-16T12:27:27.239 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:26 smithi098 podman[92923]: 2024-03-16 12:27:26.979269312 +0000 UTC m=+0.566173287 container remove 7b3568f197c27a07884d85e8a5c192b1258c80e75d5155047f5946af84f42d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T12:27:27.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T12:27:27.532 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:27 smithi098 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.g.service: Deactivated successfully. 2024-03-16T12:27:27.532 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:27 smithi098 systemd[1]: Stopped Ceph mds.g for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:27:27.532 INFO:journalctl@ceph.mds.g.smithi098.stdout:Mar 16 12:27:27 smithi098 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.g.service: Consumed 2.146s CPU time. 2024-03-16T12:27:27.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:27.564 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T12:27:27.564 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T12:27:27.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.j 2024-03-16T12:27:27.990 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:27 smithi098 systemd[1]: Stopping Ceph mds.j for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:28.490 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j[72290]: 2024-03-16T12:27:28.194+0000 7f2157200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:28.490 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:28 smithi098 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j[72290]: 2024-03-16T12:27:28.194+0000 7f2157200700 -1 mds.j *** got signal Terminated *** 2024-03-16T12:27:32.472 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:32 smithi098 podman[93028]: 2024-03-16 12:27:32.212270422 +0000 UTC m=+4.142726435 container died b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T12:27:32.740 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:32 smithi098 podman[93028]: 2024-03-16 12:27:32.598385574 +0000 UTC m=+4.528841626 container cleanup b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T12:27:32.740 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:32 smithi098 bash[93028]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j 2024-03-16T12:27:33.240 INFO:journalctl@ceph.mds.j.smithi098.stdout:Mar 16 12:27:32 smithi098 podman[93042]: 2024-03-16 12:27:32.807542663 +0000 UTC m=+0.582513575 container remove b245bc00bcc29b4dc42db1db5c1f3102c942ca9f701f59fa338aa4a7c7599107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:27:33.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T12:27:33.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:33.409 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T12:27:33.409 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T12:27:33.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.b 2024-03-16T12:27:33.709 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:33 smithi153 systemd[1]: Stopping Ceph mds.b for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:34.001 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: 2024-03-16T12:27:33.730+0000 7f3d41fe2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:34.001 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:33 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b[67655]: 2024-03-16T12:27:33.730+0000 7f3d41fe2700 -1 mds.b *** got signal Terminated *** 2024-03-16T12:27:36.993 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:36 smithi153 podman[71788]: 2024-03-16 12:27:36.732319911 +0000 UTC m=+3.109687142 container died 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T12:27:37.251 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:37 smithi153 podman[71788]: 2024-03-16 12:27:37.12225074 +0000 UTC m=+3.499617951 container cleanup 7767e6eb625797eef86ef06bf470533880722f6f6ae44d9b0815138052767544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T12:27:37.251 INFO:journalctl@ceph.mds.b.smithi153.stdout:Mar 16 12:27:37 smithi153 bash[71788]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-b 2024-03-16T12:27:37.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T12:27:37.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:37.556 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T12:27:37.556 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T12:27:37.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.e 2024-03-16T12:27:37.878 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:37 smithi153 systemd[1]: Stopping Ceph mds.e for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:38.251 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: 2024-03-16T12:27:37.900+0000 7ff0bfa74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:38.251 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:37 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e[67032]: 2024-03-16T12:27:37.900+0000 7ff0bfa74700 -1 mds.e *** got signal Terminated *** 2024-03-16T12:27:41.001 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:40 smithi153 podman[71921]: 2024-03-16 12:27:40.729101114 +0000 UTC m=+2.935504671 container died 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T12:27:41.313 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 podman[71921]: 2024-03-16 12:27:41.15051509 +0000 UTC m=+3.356918637 container cleanup 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:27:41.313 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 bash[71921]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e 2024-03-16T12:27:41.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T12:27:41.568 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 podman[71935]: 2024-03-16 12:27:41.335085677 +0000 UTC m=+0.600999933 container remove 77f5946986a9642d49ff1146cee3fa248cc3355c3e843038bf307f333c240e67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:27:41.568 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.e.service: Deactivated successfully. 2024-03-16T12:27:41.568 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 systemd[1]: Stopped Ceph mds.e for e9cfddf2-e38d-11ee-95c9-87774f69a715. 2024-03-16T12:27:41.568 INFO:journalctl@ceph.mds.e.smithi153.stdout:Mar 16 12:27:41 smithi153 systemd[1]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.e.service: Consumed 1.316s CPU time. 2024-03-16T12:27:41.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:41.579 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T12:27:41.580 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T12:27:41.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.h 2024-03-16T12:27:41.907 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:41 smithi153 systemd[1]: Stopping Ceph mds.h for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:42.251 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: 2024-03-16T12:27:41.929+0000 7f49c641d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:42.251 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:41 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h[67347]: 2024-03-16T12:27:41.929+0000 7f49c641d700 -1 mds.h *** got signal Terminated *** 2024-03-16T12:27:45.001 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:44 smithi153 podman[72043]: 2024-03-16 12:27:44.7299922 +0000 UTC m=+2.911384997 container died b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T12:27:45.458 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:45 smithi153 podman[72043]: 2024-03-16 12:27:45.19583383 +0000 UTC m=+3.377226558 container cleanup b29b1ba93086eec47ab80e9b772b2e6ddc1b8812e9020e0aead649f42d6e64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:27:45.458 INFO:journalctl@ceph.mds.h.smithi153.stdout:Mar 16 12:27:45 smithi153 bash[72043]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-h 2024-03-16T12:27:45.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T12:27:45.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:45.638 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T12:27:45.638 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T12:27:45.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.k 2024-03-16T12:27:45.969 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:45 smithi153 systemd[1]: Stopping Ceph mds.k for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:46.251 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k[66712]: 2024-03-16T12:27:45.990+0000 7f4d62729700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:46.251 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:45 smithi153 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k[66712]: 2024-03-16T12:27:45.990+0000 7f4d62729700 -1 mds.k *** got signal Terminated *** 2024-03-16T12:27:48.911 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:48 smithi153 podman[72164]: 2024-03-16 12:27:48.674714662 +0000 UTC m=+2.792635218 container died 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:27:49.251 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:49 smithi153 podman[72164]: 2024-03-16 12:27:49.049272403 +0000 UTC m=+3.167192950 container cleanup 634be9d16f5d57f38624a037070989751adb093a9a6d78d7ed41add1fd671a55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:27:49.251 INFO:journalctl@ceph.mds.k.smithi153.stdout:Mar 16 12:27:49 smithi153 bash[72164]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-k 2024-03-16T12:27:49.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T12:27:49.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:49.487 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T12:27:49.487 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T12:27:49.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.c 2024-03-16T12:27:49.817 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:49 smithi173 systemd[1]: Stopping Ceph mds.c for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:50.076 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: 2024-03-16T12:27:50.043+0000 7f98b69a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:50.076 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:50 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c[67204]: 2024-03-16T12:27:50.043+0000 7f98b69a5700 -1 mds.c *** got signal Terminated *** 2024-03-16T12:27:53.005 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:52 smithi173 podman[71762]: 2024-03-16 12:27:52.72025195 +0000 UTC m=+3.023195272 container died edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T12:27:53.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:53 smithi173 podman[71762]: 2024-03-16 12:27:53.02760339 +0000 UTC m=+3.330546678 container cleanup edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T12:27:53.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:53 smithi173 bash[71762]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c 2024-03-16T12:27:53.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Mar 16 12:27:53 smithi173 podman[71788]: 2024-03-16 12:27:53.253913731 +0000 UTC m=+0.527112757 container remove edfdbe6d26ed16c21e947b24ea11736fb9382a1c7412ac7d705839db77902b35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:27:53.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T12:27:53.401 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:53.401 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T12:27:53.402 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T12:27:53.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.f 2024-03-16T12:27:53.718 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:53 smithi173 systemd[1]: Stopping Ceph mds.f for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:54.067 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: 2024-03-16T12:27:53.740+0000 7f48a271a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:54.067 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:53 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f[66898]: 2024-03-16T12:27:53.740+0000 7f48a271a700 -1 mds.f *** got signal Terminated *** 2024-03-16T12:27:56.994 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:56 smithi173 podman[71896]: 2024-03-16 12:27:56.732680072 +0000 UTC m=+3.093893019 container died 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T12:27:57.317 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:57 smithi173 podman[71896]: 2024-03-16 12:27:57.098711163 +0000 UTC m=+3.459924103 container cleanup 31d8c66a952f3832b1e7c01a78d29121fff71e6e7049d6025e5cdf47e428824b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T12:27:57.317 INFO:journalctl@ceph.mds.f.smithi173.stdout:Mar 16 12:27:57 smithi173 bash[71896]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-f 2024-03-16T12:27:57.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T12:27:57.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:27:57.531 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T12:27:57.531 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T12:27:57.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.i 2024-03-16T12:27:58.067 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:27:57 smithi173 systemd[1]: Stopping Ceph mds.i for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:27:58.067 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:27:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: 2024-03-16T12:27:57.862+0000 7f6bcdf75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:27:58.067 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:27:57 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i[67512]: 2024-03-16T12:27:57.862+0000 7f6bcdf75700 -1 mds.i *** got signal Terminated *** 2024-03-16T12:28:00.817 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:28:00 smithi173 podman[72016]: 2024-03-16 12:28:00.735537648 +0000 UTC m=+2.965443510 container died 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:28:01.757 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:28:01 smithi173 podman[72016]: 2024-03-16 12:28:01.370111883 +0000 UTC m=+3.600017724 container cleanup 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:28:01.757 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:28:01 smithi173 bash[72016]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i 2024-03-16T12:28:01.757 INFO:journalctl@ceph.mds.i.smithi173.stdout:Mar 16 12:28:01 smithi173 podman[72030]: 2024-03-16 12:28:01.562988401 +0000 UTC m=+0.829901548 container remove 79d40dcf9895c5f68152fdbb3984654ddaaf8495e1e2a8eaca346b935f4a54d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T12:28:01.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T12:28:01.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:28:01.808 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T12:28:01.808 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T12:28:01.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.l 2024-03-16T12:28:02.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:01 smithi173 systemd[1]: Stopping Ceph mds.l for e9cfddf2-e38d-11ee-95c9-87774f69a715... 2024-03-16T12:28:02.567 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l[66588]: 2024-03-16T12:28:02.148+0000 7f2ea477c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:28:02.567 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:02 smithi173 ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l[66588]: 2024-03-16T12:28:02.148+0000 7f2ea477c700 -1 mds.l *** got signal Terminated *** 2024-03-16T12:28:06.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:05 smithi173 podman[72141]: 2024-03-16 12:28:05.716311771 +0000 UTC m=+3.671096469 container died 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T12:28:06.504 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:06 smithi173 podman[72141]: 2024-03-16 12:28:06.224448292 +0000 UTC m=+4.179232972 container cleanup 01869771ba2d402ab1ce5a8aa53a57692df2c7cb8165b2f53225f5460c7d73be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T12:28:06.504 INFO:journalctl@ceph.mds.l.smithi173.stdout:Mar 16 12:28:06 smithi173 bash[72141]: ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715-mds-l 2024-03-16T12:28:06.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e9cfddf2-e38d-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T12:28:06.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:28:06.698 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T12:28:06.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:28:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:11.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:28:11.933 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:14.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:28:14.665 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:17.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:28:17.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:28:17.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:28:17.316 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T12:28:17.318 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi098/crash 2024-03-16T12:28:17.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:28:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:28:17.447 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:28:17.453 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi153/crash 2024-03-16T12:28:17.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:28:17.492 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:28:17.492 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:28:17.494 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi173/crash 2024-03-16T12:28:17.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:28:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:28:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:28:17.530 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T12:28:17.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T12:28:17.638 INFO:teuthology.orchestra.run.smithi098.stdout:2024-03-16T12:11:50.277153+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:28:17.639 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T12:28:17.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[SEC\]' /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T12:28:17.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]' /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T12:28:17.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[WRN\]' /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T12:28:17.909 INFO:teuthology.orchestra.run.smithi098.stdout:2024-03-16T12:11:50.277153+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:28:17.910 INFO:tasks.cephadm:Compressing logs... 2024-03-16T12:28:17.910 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:28:17.914 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:28:17.917 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:28:17.944 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:28:17.944 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-03-16T12:28:17.944 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T12:28:17.945 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:28:17.950 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T12:28:17.950 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.l.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T12:28:17.950 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T12:28:17.950 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T12:28:17.950 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.z.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.6% 94.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T12:28:17.951 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T12:28:17.955 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.2.log: 86.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:28:17.955 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T12:28:17.955 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:28:17.955 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T12:28:17.959 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:28:17.959 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-03-16T12:28:17.959 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.b.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2%/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.7.log: -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.y.log: 92.3% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T12:28:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:28:17.961 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:28:17.961 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:28:17.963 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log: 86.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:28:17.963 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T12:28:17.966 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:28:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T12:28:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:28:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T12:28:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T12:28:17.977 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:28:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:28:17.980 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T12:28:17.981 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T12:28:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T12:28:17.983 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.j.log: 93.2%gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T12:28:17.984 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T12:28:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.3.log: 90.1% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:28:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T12:28:17.987 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:28:17.987 INFO:teuthology.orchestra.run.smithi098.stderr: 94.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T12:28:17.987 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-16T12:28:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:28:17.989 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.c.log: 94.2% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T12:28:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:28:17.990 INFO:teuthology.orchestra.run.smithi098.stderr: 82.9% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:28:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T12:28:17.991 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T12:28:17.993 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T12:28:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T12:28:17.994 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:28:17.994 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T12:28:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T12:28:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-16T12:28:18.004 INFO:teuthology.orchestra.run.smithi098.stderr: 94.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T12:28:18.007 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5%/var/log/ceph/ceph-client.admin.log: -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T12:28:18.009 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:28:18.009 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-16T12:28:18.017 INFO:teuthology.orchestra.run.smithi173.stderr: 93.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T12:28:18.173 INFO:teuthology.orchestra.run.smithi153.stderr: 92.9% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T12:28:18.213 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T12:28:18.214 INFO:teuthology.orchestra.run.smithi173.stderr: 91.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T12:28:18.237 INFO:teuthology.orchestra.run.smithi153.stderr: 91.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T12:28:18.488 INFO:teuthology.orchestra.run.smithi173.stderr: 92.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T12:28:18.594 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T12:28:18.614 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T12:28:18.643 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T12:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr: 90.5% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T12:28:18.685 INFO:teuthology.orchestra.run.smithi153.stderr: 92.3% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T12:28:18.861 INFO:teuthology.orchestra.run.smithi173.stderr: 93.7% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T12:28:19.042 INFO:teuthology.orchestra.run.smithi173.stderr: 93.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T12:28:19.101 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T12:28:19.150 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T12:28:19.152 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-16T12:28:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m1.220s 2024-03-16T12:28:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m4.231s 2024-03-16T12:28:19.152 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.161s 2024-03-16T12:28:19.177 INFO:teuthology.orchestra.run.smithi173.stderr: 93.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T12:28:19.179 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-16T12:28:19.179 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m1.247s 2024-03-16T12:28:19.179 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m4.067s 2024-03-16T12:28:19.179 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.168s 2024-03-16T12:28:19.221 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T12:28:19.349 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/e9cfddf2-e38d-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T12:28:19.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-16T12:28:19.354 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.408s 2024-03-16T12:28:19.354 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m4.479s 2024-03-16T12:28:19.354 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.451s 2024-03-16T12:28:19.354 INFO:tasks.cephadm:Archiving logs... 2024-03-16T12:28:19.354 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi098/log 2024-03-16T12:28:19.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:28:20.052 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi153/log 2024-03-16T12:28:20.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:28:20.681 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi173/log 2024-03-16T12:28:20.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:28:21.292 INFO:tasks.cephadm:Removing cluster... 2024-03-16T12:28:21.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force 2024-03-16T12:28:21.604 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:22.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force 2024-03-16T12:28:22.375 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:22.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e9cfddf2-e38d-11ee-95c9-87774f69a715 --force 2024-03-16T12:28:22.830 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: e9cfddf2-e38d-11ee-95c9-87774f69a715 2024-03-16T12:28:23.086 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T12:28:23.086 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:28:23.142 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:28:23.158 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:28:23.173 INFO:tasks.cephadm:Teardown complete 2024-03-16T12:28:23.174 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T12:28:23.184 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T12:28:23.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:28:23.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:28:23.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:28:23.352 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T12:28:23.363 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T12:28:23.364 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:28:23.371 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:28:23.374 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:28:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:28:23.388 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +1051us[+1051us] +/- 114ms 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -140us[ -146us] +/- 51ms 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -555us[ -562us] +/- 92ms 2024-03-16T12:28:23.390 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:28:23.392 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:28:23.392 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-16T12:28:23.392 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +1037us[+1002us] +/- 114ms 2024-03-16T12:28:23.392 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -131us[ -166us] +/- 51ms 2024-03-16T12:28:23.393 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -559us[ -559us] +/- 92ms 2024-03-16T12:28:23.393 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:28:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:28:23.427 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:28:23.427 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-16T12:28:23.428 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +1040us[+1040us] +/- 114ms 2024-03-16T12:28:23.428 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -137us[ -172us] +/- 51ms 2024-03-16T12:28:23.428 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -548us[ -548us] +/- 92ms 2024-03-16T12:28:23.428 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:28:23.429 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T12:28:23.439 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T12:28:23.440 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T12:28:23.449 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:28:23.678 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:28:23.748 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:28:23.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:28:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:23.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:28:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:23.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:28:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:23.982 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T12:28:23.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T12:28:24.003 INFO:teuthology.task.internal:Duration was 1671.665501 seconds 2024-03-16T12:28:24.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T12:28:24.012 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T12:28:24.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:28:24.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:28:24.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:28:24.063 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:28:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:28:24.348 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T12:28:24.348 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-16T12:28:24.348 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:28:24.417 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-16T12:28:24.418 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:28:24.447 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-16T12:28:24.447 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:28:24.472 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T12:28:24.472 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:28:24.480 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:28:24.489 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:28:24.572 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T12:28:24.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:28:24.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:28:24.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:28:25.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T12:28:25.077 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T12:28:25.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:28:25.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:28:25.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:28:25.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T12:28:25.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.262 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-16T12:28:25.284 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-16T12:28:25.337 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-16T12:28:25.391 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:25.439 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:25.453 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:28:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:28:25.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T12:28:25.479 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T12:28:25.480 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi098 2024-03-16T12:28:25.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:28:25.588 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi153 2024-03-16T12:28:25.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:28:25.628 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434/remote/smithi173 2024-03-16T12:28:25.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:28:25.667 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T12:28:25.667 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:28:25.675 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:28:25.678 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:28:25.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T12:28:25.737 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T12:28:25.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T12:28:25.750 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T12:28:25.750 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:28:25.758 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:28:25.761 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:28:25.772 INFO:teuthology.orchestra.run.smithi153.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:28 /home/ubuntu/cephtest 2024-03-16T12:28:25.774 INFO:teuthology.orchestra.run.smithi173.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:28 /home/ubuntu/cephtest 2024-03-16T12:28:25.807 INFO:teuthology.orchestra.run.smithi098.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:28 /home/ubuntu/cephtest 2024-03-16T12:28:25.810 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T12:28:25.822 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T12:28:25.909 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T12:28:25.936 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-16T12:28:25.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:28:25.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.815087', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwWd8rBTHCIAQBEQi5HXxWn8S2bWvP1M405ygUBRyMEBjX1B095iiaWEDws2QLjTsUg2K4LqmgJmcr/5vhlN7U='} 2024-03-16T12:28:25.962 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-16T12:28:25.962 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:28:25.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.817083', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEPN8BI7vA1jD+GZIPR75ZCizztI2zUCeQIKFnkor63kh2dMpO8kJc4da7TqQdaovLvcPKIWXe7OOdRFsKypyE='} 2024-03-16T12:28:25.989 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-16T12:28:25.989 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:28:26.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602434', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:51:28.818824', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYtuJvBvi2iHHwcVzQkrsBbGU4azcveJmHAbJYTZku90xWNJ0CCR9f4622ZojTPmiAex4NUwRK3sUfhxq18BFg='} 2024-03-16T12:28:26.021 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-16T12:28:26.021 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:28:26.046 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-16T12:28:26.046 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:26.063 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-16T12:28:26.064 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:28:26.074 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-16T12:28:26.085 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-16T12:28:26.085 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:26.100 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-16T12:28:26.101 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:28:26.110 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-16T12:28:26.124 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-16T12:28:26.124 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:26.147 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-16T12:28:30.076 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:30.112 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:30.148 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:30.186 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-16T12:28:30.221 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-16T12:28:30.258 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-16T12:28:34.190 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:34.222 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:34.259 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:34.300 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-16T12:28:34.331 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-16T12:28:34.370 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-16T12:28:34.401 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-16T12:28:34.471 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-16T12:28:38.336 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:28:38.445 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-16T12:28:38.546 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-16T12:28:38.652 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} duration: 1671.6655011177063 failure_reason: '"2024-03-16T12:11:50.277153+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-16T12:28:38.652 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T12:28:38.737 INFO:teuthology.run:FAIL